etcd stopped accepts new connections on port 2380 after a network temporary error

[zwtop]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

etcd stopped accepts new connections on port 2380 after a network temporary error.

What did you expect to happen?

etcd should always accepts new connections on port 2380 after a network temporary error, or log an error and exit.

How can we reproduce it (as minimally and precisely as possible)?

1. Run etcd with a small nofile limit

$ prlimit --nofile=160:160 -- ./etcd-v3.6.1-linux-amd64/etcd --peer-auto-tls --listen-peer-urls=https://localhost:2380

2. Generate a large number of connections to port 2380

$ cat > main.go << EOF
package main

import "net"

func main() {
        for n := 0; n < 200; n++ {
                net.Dial("tcp", "127.0.0.1:2380")
        }
}
EOF

$ go run main.go

After the above steps, etcd stopped accepts new connections on port 2380.

The Recv-Q on port 2380 would never handled by etcd.

$ netstat -ltn4
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN
tcp       53      0 127.0.0.1:2380          0.0.0.0:*               LISTEN

Anything else we need to know?

I think the problem happens here:

1. tlsListener return on any accept error:
   https://github.com/etcd-io/etcd/blob/release-3.6/client/pkg/transport/listener_tls.go#L126-L130

2. cMux continue accept on the Temporary error:
   https://github.com/soheilhy/cmux/blob/v0.1.5/cmux.go#L248-L250

3. Peer port 2380 use the cMux and tlsListener, on any network temporary errors, tlsListener stop accept but cMux continue accept. So the etcd stopped accept new connections on port 2380 without any logs.

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.6.1
Git SHA: a4708be
Go Version: go1.23.10
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.6.1
API version: 3.6

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output