metadata-2 #2273

thatcher-gaming · 2025-06-11T17:14:23Z

adds support for most of the draft/metadata-2 spec

what's new:

a new draft/metadata-2 cap
the metadata command, with SET, GET, CLEAR, LIST, SUB, UNSUB, SUBS & SYNC
a metadata section in the config

what's missing:

before-connect
visibility controls. as of right now everything is *
any attempt at rate limiting
postponed synchronisation

this is a draft because i still need to write some tests but i thought i'd open a pr so someone can tell me if i've done something egregiously wrong (this is my first time doing anything particularly substantial in go (!!!!))

slingamn · 2025-06-11T19:26:34Z

Thank you so much, this is delightful!

I'll let @progval weigh in on correctness w.r.t. the specification.

I have a brief implementation note: don't write any new code referencing buntdb. Store everything (the metadata itself, the subscription list) as members of the Client (not Session) and Channel structs, protected by the stateMutex locks in those structs. Use those members as the source of truth for all runtime reads and writes.

I am in the process of refactoring the account system to have asynchronous persistence, but the corresponding refactor for channels is already done (#2028), so if you want to persist channel metadata, here's how you do it:

Add a Metadata member to RegisteredChannel (this is the struct that gets serialized to the database for registered channels). Add fields that copy the data to (*Channel).applyRegInfo and (*Channel).ExportRegistration (the first copies from RegisteredChannel to Channel, the second in the reverse direction).
After updating channel metadata, call channel.MarkDirty(IncludeAllAttrs).

As for clients/accounts, don't write any persistence code yet, I will add it as part of the accounts refactor :-)

slingamn · 2025-06-11T19:30:05Z

Ah, I think I was wrong earlier: if the expectation is that metadata subscriptions have to be refreshed by the client on every reconnection, then the correct place to store them is indeed the Session object (and they don't persist).

irc/metadata.go

irc/config.go

progval

Very nice, thanks.

First wave of reviews from me, needed so I can run the rest of my tests.

irc/handlers.go

progval

Another round of reviews before I go to bed

irc/handlers.go

progval · 2025-06-12T20:01:04Z

irc/handlers.go

+			noKeyPerms(key)
+			return
+		}
+


@slingamn Should we check the key is small enough that it's not going to be truncated when sent to other clients? (METADATA * SET display-name :abc is shorter than :irc.example.com METADATA user1 display-name * :abc)

Is this addressed in the spec at all?

I support adding a preemptive sanity check here, but we probably shouldn't do arithmetic with the server name length, I think it should be more like "key + value are less than 350 bytes"

of course not, this is IRC :)

thatcher-gaming · 2025-06-13T00:56:02Z

After updating channel metadata, call channel.MarkDirty(IncludeAllAttrs).

@slingamn This is now in, but there's a problem: calling this during set seems to make the function hang forever. I do not know why this is.

thatcher-gaming · 2025-06-13T01:05:04Z

Oh and @progval, I'm finding this bit of the spec somewhat confusing:

If the target is a channel, it also syncs the metadata for all other users in that channel

My sleep deprived brain interpreted it as "send information about the channel to each user in the channel", but I now realise it's much more likely to mean "send the information about all the channel's participants to the client". I am not a smart person and have no idea if this would actually be a common interpretation, so feel free to disregard.

slingamn

This is really nice. @progval can hook you up with the unreleased integration tests for this

slingamn · 2025-06-13T01:00:11Z

irc/metadata.go

+
+type MetadataStore = map[string]string
+
+type MetadataThing = struct {


This is dead code now....I don't think we're going to implement visibility as part of the MVP for this so we can just use map[string]string for now

slingamn · 2025-06-13T01:02:30Z

irc/metadata.go

+		members := targetChannel.Members()
+		for _, m := range members {
+			for _, s := range m.Sessions() {
+				notify.Add(s)


Move the cap check up here...

If you want to get adventurous you can actually use iter.Seq here, so the client case will still allocate a hashset of friend sessions (because we need to deduplicate), but the channel case can just iterate over Members() and then Sessions() per member and return applicable sessions without allocating, and then iter.Seq is a common interface for ranging over both of these.

irc/metadata.go

irc/client.go

irc/handlers.go

irc/client.go

irc/handlers.go

progval · 2025-06-13T04:47:59Z

Oh and @progval, I'm finding this bit of the spec somewhat confusing:

If the target is a channel, it also syncs the metadata for all other users in that channel

My sleep deprived brain interpreted it as "send information about the channel to each user in the channel", but I now realise it's much more likely to mean "send the information about all the channel's participants to the client". I am not a smart person and have no idea if this would actually be a common interpretation, so feel free to disregard.

that sounds right

thatcher-gaming · 2025-06-13T17:12:56Z

I'm 75% sure this is ready for primetime now. Let me know if something goes wrong!!

slingamn

This is getting very close to passing Val's tests

irc/utils/chunks.go

irc/handlers.go

irc/channelreg.go

gencapdefs.py

irc/client.go

irc/getters.go

slingamn · 2025-06-13T18:38:03Z

@thatcher-gaming can you add this functionality?

On joining a channel, users will get the channel’s current metadata sent to them with METADATA messages, and get the same information for all users who are in the channel. Specifically, they get that information for the keys they are subscribed to. The server may also tell them to request that information at a later time.

slingamn

Looks good! I'll leave this open while we do more testing.

slingamn · 2025-06-13T21:51:11Z

irc/getters.go

+	if session.metadataSubscriptions == nil {
+		return false
+	}


You can delete this (reads and deletes from the nil map in Go are no-ops like an empty map --- stores will panic)

slingamn · 2025-06-13T21:51:37Z

irc/getters.go

+	if session.metadataSubscriptions == nil {
+		return []string{}
+	}


delete as per above

slingamn · 2025-06-13T21:51:49Z

irc/getters.go

+	if channel.metadata == nil {
+		channel.metadata = make(map[string]string)
+	}


delete as per above

slingamn · 2025-06-13T21:53:51Z

irc/getters.go

+	if channel.metadata == nil {
+		return 0
+	}


slingamn · 2025-06-13T21:54:05Z

irc/getters.go

+	channel.stateMutex.Lock()
+
+	oldMap := channel.metadata
+	channel.metadata = make(map[string]string)


channel.metadata = nil is OK here

slingamn · 2025-06-13T21:56:03Z

irc/getters.go

+	channel.metadata = make(map[string]string)
+
+	channel.stateMutex.Unlock()
+	channel.MarkDirty(IncludeAllAttrs)


I have PTSD from #2149 so my preferred style for all locking is to unlock in a defer, something like this:

defer channel.MarkDirty(IncludeAllAttrs) channel.stateMutex.Lock() defer channel.stateMutex.Unlock() // do the write operation

slingamn · 2025-06-13T21:56:47Z

irc/getters.go

+	delete(channel.metadata, key)
+
+	channel.stateMutex.Unlock()
+	channel.MarkDirty(IncludeAllAttrs)


see note below about defer

slingamn · 2025-06-13T21:57:50Z

irc/getters.go

+	return maps.Clone(session.metadataSubscriptions)
+}
+
+func (channel *Channel) GetMetadata(key string) (string, error) {


I would return (string, bool) here (the value and ok) --- makes it clearer how to exhaustively handle the success and failure conditions

slingamn

Looks great from an implementation standpoint! I'd like to get another look from @progval on correctness w.r.t. the spec.

This also passes our updated test suite at https://github.com/progval/irctest/tree/metadata2 as mentioned.

slingamn · 2025-06-15T02:43:07Z

irc/help.go

+	},
+	"metadata": {
+		text: `METADATA <target> <subcommand> [<everything else>...]
+		


extra whitespace on this line, should be blank

slingamn · 2025-06-15T08:06:30Z

I'm going to merge this so I can work on some follow-up tasks. Thanks for the great PR, @thatcher-gaming !

thatcher-gaming · 2025-06-15T11:50:50Z

thanks for all the feedback! sorry about my slightly amateur go knowledge haha

slingamn reviewed Jun 11, 2025

View reviewed changes

irc/metadata.go Outdated Show resolved Hide resolved

slingamn reviewed Jun 11, 2025

View reviewed changes

irc/config.go Show resolved Hide resolved

progval reviewed Jun 11, 2025

View reviewed changes

irc/handlers.go Outdated Show resolved Hide resolved

irc/handlers.go Outdated Show resolved Hide resolved

irc/handlers.go Show resolved Hide resolved

irc/handlers.go Show resolved Hide resolved

progval reviewed Jun 12, 2025

View reviewed changes

slingamn reviewed Jun 13, 2025

View reviewed changes

add a bit of metadata-2 support

2275bed

thatcher-gaming force-pushed the master branch from 39ec6ae to 2275bed Compare June 13, 2025 17:10

thatcher-gaming marked this pull request as ready for review June 13, 2025 17:12

thatcher-gaming changed the title ~~WIP: metadata-2~~ metadata-2 Jun 13, 2025

enforce max-keys

db4b23b

slingamn mentioned this pull request Jun 13, 2025

follow-up issues for metadata #2274

Closed

12 tasks

slingamn reviewed Jun 13, 2025

View reviewed changes

thatcher-gaming force-pushed the master branch 2 times, most recently from d025e39 to 0cf66d7 Compare June 13, 2025 19:03

tidy up a bit

6d94aa1

thatcher-gaming force-pushed the master branch from 0cf66d7 to 6d94aa1 Compare June 13, 2025 19:06

slingamn reviewed Jun 13, 2025

View reviewed changes

thatcher-gaming added 2 commits June 13, 2025 23:24

remove unnecessary bits

c46fe12

that but again

4741a30

slingamn approved these changes Jun 15, 2025

View reviewed changes

slingamn merged commit 4dcbc48 into ergochat:master Jun 15, 2025
1 check passed

slingamn added this to the v2.17 milestone Jun 16, 2025

slingamn added IRCv3 new cap New client cap labels Jun 16, 2025


		type MetadataStore = map[string]string

		type MetadataThing = struct {

metadata-2 #2273

metadata-2 #2273

Uh oh!

Conversation

thatcher-gaming commented Jun 11, 2025

Uh oh!

slingamn commented Jun 11, 2025

Uh oh!

slingamn commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

progval left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

progval left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

progval Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thatcher-gaming commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thatcher-gaming commented Jun 13, 2025

Uh oh!

slingamn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

progval commented Jun 13, 2025

Uh oh!

thatcher-gaming commented Jun 13, 2025

Uh oh!

slingamn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slingamn commented Jun 13, 2025

Uh oh!

slingamn left a comment

Choose a reason for hiding this comment

Uh oh!

progval Jun 12, 2025 •

edited

Loading

thatcher-gaming commented Jun 13, 2025 •

edited

Loading