What happened:
secondary module does not respect the reload signal. For example you have 2 servers in your Corefile, both with secondary towards some servers. You remove the 2nd virtual server from your Corefile and reload coreDNS. Efforts to get the Zone which the 2nd virtual server had in its configuration is still happening despite it is now gone from the Corefile and the service reloaded

What you expected to happen:
secondary plugin shouldn't have run for the 2nd virtual server as it is gone from the Corefile

How to reproduce it (as minimally and precisely as possible):
Set in your Corefile 2 servers where both are getting a Zone

# Common plugins
(common) {
    bind 127.0.0.1
    errors
    log
    cache
    acl {
        allow type AXFR A AAAA net 127.0.0.1/32
        block
    }
    prometheus 127.0.0.1:9052
}

# DNS Resolver AGI            
(dns-resolver-1) {
    import common
    secondary {
        transfer from 10.98.4.5 10.98.4.6
    }
    transfer {
        to 127.0.0.1
    }
}
agi.appgate.com:9054 {
        health
        import dns-resolver-1
}

# DNS Resolver 2            
(dns-resolver-2) {
    import common
    secondary {
        transfer from 1.1.1.1
    }
    transfer {
        to 127.0.0.1
    }
}
example.com:9055 {
        import dns-resolver-2
}

Remove one of them let's say the example.com:9055 and reload coreDNS

In the logs you see that coreDNS is still trying to download the zone that was declared in the removed server

[ERROR] plugin/file: Failed to transfer `example.com.' from "1.1.1.1:53": dns: bad xfr rcode: 5
[WARNING] plugin/secondary: All 'example.com.' masters failed to transfer, retrying in 10s: dns: bad xfr rcode: 5

A restart clears the issue

Anything else we need to know?:
Maybe related to #6703 ?

Environment:
coreDNS v1.11.3
Ubuntu 22.04.4 LTS