Support of address reuse in server reload #6

glebkin · 2024-09-23T19:22:29Z

We're preparing changes to CoreDNS to support SO_REUSEPORT option, which allowes user to create multiple listening sockets to the same port - coredns/coredns#6882.

Currently we may receive unexpected behaviour during server reload because caddy handles file descriptors as one fd per address.

Before reload:

ss -ulpn | grep 54
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=163806,fd=13))       
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=163806,fd=14))       
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=163806,fd=17))       
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=163806,fd=12))

cat /proc/163806/net/udp6
  sl  local_address                         remote_address                        st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 272953 2 ffff9ae06842d940 0
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 272955 2 ffff9ae068429a40 0
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 272957 2 ffff9ae06842de80 0
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 272959 2 ffff9ae068429500 0

After reload:

ss -ulpn | grep 54
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=163806,fd=31),("coredns",pid=163806,fd=28),("coredns",pid=163806,fd=26),("coredns",pid=163806,fd=25))

cat /proc/163806/net/udp6
  sl  local_address                         remote_address                        st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 272959 2 ffff9ae068429500 0

So this change in caddy assumes that we may get multiple servers listening the same address.

Update server reload logic to support multiple servers having the same address Signed-off-by: Gleb Kogtev <[email protected]>

glebkin · 2024-09-23T20:00:04Z

It won’t work properly in case of multiple servers with different addresses. I’ll update this PR soon.

Signed-off-by: Gleb Kogtev <[email protected]>

glebkin · 2024-09-24T07:16:17Z

Fixed issue with different addresses.
Example configuration:

.:55 {
    whoami
}

.:52 {
    reuseport 3
    whoami
}

.:54 {
    reuseport 5
    whoami
    reload 2s
}

Start CoreDNS server:

ss -ulpn | grep coredns
UNCONN 0      0                  *:52              *:*    users:(("coredns",pid=228093,fd=11))       
UNCONN 0      0                  *:52              *:*    users:(("coredns",pid=228093,fd=13))       
UNCONN 0      0                  *:52              *:*    users:(("coredns",pid=228093,fd=15))       
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=228093,fd=19))       
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=228093,fd=21))       
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=228093,fd=23))       
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=228093,fd=18))       
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=228093,fd=26))       
UNCONN 0      0                  *:55              *:*    users:(("coredns",pid=228093,fd=9))

cat /proc/228093/net/udp6 
  sl  local_address                         remote_address                        st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops
21031: 00000000000000000000000000000000:0034 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347196 2 ffff9ae057f20540 0
21031: 00000000000000000000000000000000:0034 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347198 2 ffff9ae057f20000 0
21031: 00000000000000000000000000000000:0034 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347200 2 ffff9ae057f224c0 0
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347202 2 ffff9ae057f24980 0
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347204 2 ffff9ae057f25940 0
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347206 2 ffff9ae057f21a40 0
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347208 2 ffff9ae057f239c0 0
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347210 2 ffff9ae057f24440 0
21034: 00000000000000000000000000000000:0037 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347194 2 ffff9ae057f26900 0

Test DNS requests:

Server - :52

dnsperf -d test.txt -s 127.0.0.1 -p 52 -c 30 -l 10 -T 10
DNS Performance Testing Tool
Version 2.9.0

[Status] Command line: dnsperf -d test.txt -s 127.0.0.1 -p 52 -c 30 -l 10 -T 10
[Status] Sending queries (to 127.0.0.1:52)
[Status] Started at: Tue Sep 24 07:11:19 2024
[Status] Stopping after 10.000000 seconds
[Status] Testing complete (time limit)

Statistics:

  Queries sent:         1666360
  Queries completed:    1666360 (100.00%)
  Queries lost:         0 (0.00%)

  Response codes:       NOERROR 1666360 (100.00%)
  Average packet size:  request 33, response 103
  Run time (s):         10.006079
  Queries per second:   166534.763517

  Average Latency (s):  0.000579 (min 0.000019, max 0.010067)
  Latency StdDev (s):   0.000486

Server - :54

dnsperf -d test.txt -s 127.0.0.1 -p 54 -c 30 -l 10 -T 10
DNS Performance Testing Tool
Version 2.9.0

[Status] Command line: dnsperf -d test.txt -s 127.0.0.1 -p 54 -c 30 -l 10 -T 10
[Status] Sending queries (to 127.0.0.1:54)
[Status] Started at: Tue Sep 24 07:11:49 2024
[Status] Stopping after 10.000000 seconds
[Status] Testing complete (time limit)

Statistics:

  Queries sent:         2026903
  Queries completed:    2026903 (100.00%)
  Queries lost:         0 (0.00%)

  Response codes:       NOERROR 2026903 (100.00%)
  Average packet size:  request 33, response 103
  Run time (s):         10.010267
  Queries per second:   202482.411308

  Average Latency (s):  0.000465 (min 0.000019, max 0.019302)
  Latency StdDev (s):   0.000584

Server - :55

dnsperf -d test.txt -s 127.0.0.1 -p 55 -c 30 -l 10 -T 10
DNS Performance Testing Tool
Version 2.9.0

[Status] Command line: dnsperf -d test.txt -s 127.0.0.1 -p 55 -c 30 -l 10 -T 10
[Status] Sending queries (to 127.0.0.1:55)
[Status] Started at: Tue Sep 24 07:12:32 2024
[Status] Stopping after 10.000000 seconds
[Status] Testing complete (time limit)

Statistics:

  Queries sent:         501079
  Queries completed:    501079 (100.00%)
  Queries lost:         0 (0.00%)

  Response codes:       NOERROR 501079 (100.00%)
  Average packet size:  request 33, response 103
  Run time (s):         10.000853
  Queries per second:   50103.626161

  Average Latency (s):  0.001948 (min 0.000026, max 0.017732)
  Latency StdDev (s):   0.001226

Increse reuseport for :52 and decrease for :54 using reload plugin:

.:55 {
    whoami
}

.:52 {
    reuseport 4
    whoami
}

.:54 {
    reuseport 2
    whoami
    reload 2s
}

Check using sockets:

ss -ulpn | grep coredns
UNCONN 0      0                  *:52              *:*    users:(("coredns",pid=228093,fd=50))       
UNCONN 0      0                  *:52              *:*    users:(("coredns",pid=228093,fd=54))       
UNCONN 0      0                  *:52              *:*    users:(("coredns",pid=228093,fd=52))       
UNCONN 0      0                  *:52              *:*    users:(("coredns",pid=228093,fd=55))       
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=228093,fd=58))       
UNCONN 0      0                  *:54              *:*    users:(("coredns",pid=228093,fd=60))       
UNCONN 0      0                  *:55              *:*    users:(("coredns",pid=228093,fd=48))

cat /proc/228093/net/udp6 
  sl  local_address                         remote_address                        st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops
21031: 00000000000000000000000000000000:0034 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347196 2 ffff9ae057f20540 0
21031: 00000000000000000000000000000000:0034 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347198 2 ffff9ae057f20000 0
21031: 00000000000000000000000000000000:0034 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347200 2 ffff9ae057f224c0 0
21031: 00000000000000000000000000000000:0034 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 338594 2 ffff9ae05a49b480 0
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347202 2 ffff9ae057f24980 0
21033: 00000000000000000000000000000000:0036 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347210 2 ffff9ae057f24440 0
21034: 00000000000000000000000000000000:0037 00000000000000000000000000000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 347194 2 ffff9ae057f26900 0

Re-run tests:
Server :52

dnsperf -d test.txt -s 127.0.0.1 -p 52 -c 30 -l 10 -T 10
DNS Performance Testing Tool
Version 2.9.0

[Status] Command line: dnsperf -d test.txt -s 127.0.0.1 -p 52 -c 30 -l 10 -T 10
[Status] Sending queries (to 127.0.0.1:52)
[Status] Started at: Tue Sep 24 07:14:52 2024
[Status] Stopping after 10.000000 seconds
[Status] Testing complete (time limit)

Statistics:

  Queries sent:         1852177
  Queries completed:    1852177 (100.00%)
  Queries lost:         0 (0.00%)

  Response codes:       NOERROR 1852177 (100.00%)
  Average packet size:  request 33, response 103
  Run time (s):         10.017110
  Queries per second:   184901.333818

  Average Latency (s):  0.000517 (min 0.000019, max 0.018841)
  Latency StdDev (s):   0.000517

Server :54

dnsperf -d test.txt -s 127.0.0.1 -p 54 -c 30 -l 10 -T 10
DNS Performance Testing Tool
Version 2.9.0

[Status] Command line: dnsperf -d test.txt -s 127.0.0.1 -p 54 -c 30 -l 10 -T 10
[Status] Sending queries (to 127.0.0.1:54)
[Status] Started at: Tue Sep 24 07:15:23 2024
[Status] Stopping after 10.000000 seconds
[Status] Testing complete (time limit)

Statistics:

  Queries sent:         1143470
  Queries completed:    1143470 (100.00%)
  Queries lost:         0 (0.00%)

  Response codes:       NOERROR 1143470 (100.00%)
  Average packet size:  request 33, response 103
  Run time (s):         10.001427
  Queries per second:   114330.685011

  Average Latency (s):  0.000847 (min 0.000018, max 0.013910)
  Latency StdDev (s):   0.000837

Server :55

dnsperf -d test.txt -s 127.0.0.1 -p 55 -c 30 -l 10 -T 10
DNS Performance Testing Tool
Version 2.9.0

[Status] Command line: dnsperf -d test.txt -s 127.0.0.1 -p 55 -c 30 -l 10 -T 10
[Status] Sending queries (to 127.0.0.1:55)
[Status] Started at: Tue Sep 24 07:15:56 2024
[Status] Stopping after 10.000000 seconds
[Status] Testing complete (time limit)

Statistics:

  Queries sent:         503090
  Queries completed:    503090 (100.00%)
  Queries lost:         0 (0.00%)

  Response codes:       NOERROR 503090 (100.00%)
  Average packet size:  request 33, response 103
  Run time (s):         10.001395
  Queries per second:   50301.982873

  Average Latency (s):  0.001934 (min 0.000029, max 0.022010)
  Latency StdDev (s):   0.001226

johnbelamaric · 2024-10-03T14:15:20Z

Can you add tests for this?

glebkin · 2024-10-06T17:04:47Z

@johnbelamaric do you expect tests for this piece of code:

	restartFds := make(map[string][]restartTriple)
	for _, s := range i.servers {
		gs, srvOk := s.server.(GracefulServer)
		ln, lnOk := s.listener.(Listener)
		pc, pcOk := s.packet.(PacketConn)
		if srvOk {
			if lnOk && pcOk {
				restartFds[gs.Address()] = append(restartFds[gs.Address()], restartTriple{server: gs, listener: ln, packet: pc})
				continue
			}
			if lnOk {
				restartFds[gs.Address()] = append(restartFds[gs.Address()], restartTriple{server: gs, listener: ln})
				continue
			}
			if pcOk {
				restartFds[gs.Address()] = append(restartFds[gs.Address()], restartTriple{server: gs, packet: pc})
				continue
			}
		}
	}

or for this:

// If this is a reload and s is a GracefulServer,
		// reuse the listener for a graceful restart.
		if gs, ok := s.(GracefulServer); ok && restartFds != nil {
			addr := gs.Address()
			// Multiple servers may use the same addr (SO_REUSEPORT option set), so it's important to ensure
			// that we don't reuse the same listener/packetconn.
			// We'll create new listeners in case there are no more available triples for the same address.
			if triples, ok := restartFds[addr]; ok && len(triples) > 0 {
				// Take first available triple
				old := triples[0]
				// Remove reused triple from restartFds
				triples[0] = triples[len(triples)-1]
				restartFds[addr] = triples[:len(triples)-1]

				// listener
				if old.listener != nil {
					file, err := old.listener.File()
					if err != nil {
						return fmt.Errorf("getting old listener file: %v", err)
					}
					ln, err = net.FileListener(file)
					if err != nil {
						return fmt.Errorf("getting file listener: %v", err)
					}
					err = file.Close()
					if err != nil {
						return fmt.Errorf("closing copy of listener file: %v", err)
					}
				}
				// packetconn
				if old.packet != nil {
					file, err := old.packet.File()
					if err != nil {
						return fmt.Errorf("getting old packet file: %v", err)
					}
					pc, err = net.FilePacketConn(file)
					if err != nil {
						return fmt.Errorf("getting file packet connection: %v", err)
					}
					err = file.Close()
					if err != nil {
						return fmt.Errorf("close copy of packet file: %v", err)
					}
				}
				ln = gs.WrapListener(ln)
			}
		}

Or for the whole restart process? It would be a bit difficult, as we're working with the real connections and file descriptors and caddy server doesn't have tests for this right now.

johnbelamaric · 2024-10-07T15:51:40Z

Yeah, I saw no existing tests. Just worries me to change the logic and expect it to just work. What about at least a CI test that does what you did manually? ie, restarts with a couple different servers/configs and makes sure the resulting sockets are correct?

Shmillerov · 2024-10-09T12:55:27Z

Hello! What do we need to do to make this CI test? We need to fix caddy in this repository, update version of caddy in the coredns repository with numsockets fix, and then make reload tests in https://github.com/coredns/ci.
Is it possible to do without merging everything to master? Do you have some process how to make CI tests for such situation, when I have everything in dev branches?

Also I have an idea how to test everything. We doesn't have tests in caddy for Restart, but we have them in CoreDNS - https://github.com/coredns/coredns/blob/master/test/reload_test.go
Using these tests we can understand that reload logic (and Restart logic) for single server works as a before.
Also I can add additional test reload+numsockets here.
What do you think, is it ok?

Shmillerov · 2024-10-21T09:20:39Z

and makes sure the resulting sockets are correct

Hello. I wrote a draft of the unit test to make sure of this - reload_test.patch

And I will say that it is impossible to test it at all.
We can retrieve a os.File from PacketConn/Listener, but this method returns a copy of the underlying file
So:

we can't compare *os.File by address
the files have a name like tcp:[::]:8080->. Therefore, the same name will be used for one port
It's impossible to get file path from os.File to compare that we are really using the same file
Fd() of files changes after using the file in net.FileListener/net.FilePacketConn
It's impossible to mock net.FileListener/net.FilePacketConn
I have not found a way to list sockets in go. Using ss or other tools directly from go seems like a bad idea.

However, I added a reload test for numsockets to the CoreDNS, there are also existing reload plugin tests there. Also we did a manual tests. So it seems we did everything we could :)

Support of address reuse in server reload

8594470

Update server reload logic to support multiple servers having the same address Signed-off-by: Gleb Kogtev <[email protected]>

glebkin mentioned this pull request Sep 23, 2024

multisocket plugin coredns/coredns#6882

Merged

Fix issue with multiple servers with different addresses

c7def4b

Signed-off-by: Gleb Kogtev <[email protected]>

Shmillerov mentioned this pull request Oct 14, 2024

coredns doesn't perform better despite having more cores coredns/coredns#5595

Closed

johnbelamaric merged commit 8de9853 into coredns:master Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support of address reuse in server reload #6

Support of address reuse in server reload #6

Uh oh!

glebkin commented Sep 23, 2024

Uh oh!

glebkin commented Sep 23, 2024

Uh oh!

glebkin commented Sep 24, 2024

Uh oh!

johnbelamaric commented Oct 3, 2024

Uh oh!

glebkin commented Oct 6, 2024

Uh oh!

johnbelamaric commented Oct 7, 2024

Uh oh!

Shmillerov commented Oct 9, 2024

Uh oh!

Shmillerov commented Oct 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Support of address reuse in server reload #6

Support of address reuse in server reload #6

Uh oh!

Conversation

glebkin commented Sep 23, 2024

Uh oh!

glebkin commented Sep 23, 2024

Uh oh!

glebkin commented Sep 24, 2024

Uh oh!

johnbelamaric commented Oct 3, 2024

Uh oh!

glebkin commented Oct 6, 2024

Uh oh!

johnbelamaric commented Oct 7, 2024

Uh oh!

Shmillerov commented Oct 9, 2024

Uh oh!

Shmillerov commented Oct 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants