Connection Queue Review
上一个帖子说当服务端的连接队列满时,服务端什么都不响应,促使客户端重传SYN segment,实际上没这么简单
下面我们通过实验说明:
在linux上启动simpletcpserver
代码: 全选
linux $ ./simpletcpserver -d 100000 -q 3 8888 # -q 3表示指定backlog为3,那么全连接队列大小则为4,-d 100000表示延迟100000秒再调用accept
在macos os上启动backlogtest
代码: 全选
macos $ ./backlogtest -c 10 linux 8888 # 启动10个线程,同时发起连接
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
在Linux上使用mytcpdump观察segments
代码: 全选
linux $ sudo ./mytcpdump port 8888
0.000 ( 0.000) IP macos.52551 > linux.8888: flags [S], seq 274874867, win 65535, options [mss 1460,ws 6,ts 990385264 0], length 0
0.000 ( 0.000) IP macos.52552 > linux.8888: flags [S], seq 3641250079, win 65535, options [mss 1460,ws 6,ts 990385264 0], length 0
0.000 ( 0.000) IP macos.52553 > linux.8888: flags [S], seq 2153714274, win 65535, options [mss 1460,ws 6,ts 990385264 0], length 0
0.000 ( 0.000) IP macos.52554 > linux.8888: flags [S], seq 99659206, win 65535, options [mss 1460,ws 6,ts 990385264 0], length 0
0.000 ( 0.000) IP macos.52555 > linux.8888: flags [S], seq 4198299130, win 65535, options [mss 1460,ws 6,ts 990385264 0], length 0
0.000 ( 0.000) IP macos.52556 > linux.8888: flags [S], seq 2660334176, win 65535, options [mss 1460,ws 6,ts 990385264 0], length 0
0.000 ( 0.000) IP macos.52557 > linux.8888: flags [S], seq 693903927, win 65535, options [mss 1460,ws 6,ts 990385264 0], length 0
0.000 ( 0.000) IP macos.52558 > linux.8888: flags [S], seq 1809403647, win 65535, options [mss 1460,ws 6,ts 990385264 0], length 0
0.000 ( 0.000) IP linux.8888 > macos.52551: flags [SA], seq 254823052, ack 274874868, win 65160, options [mss 1460,ts 1603279394 990385264,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.52552: flags [SA], seq 433081043, ack 3641250080, win 65160, options [mss 1460,ts 1603279394 990385264,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.52553: flags [SA], seq 2399418257, ack 2153714275, win 65160, options [mss 1460,ts 1603279394 990385264,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.52554: flags [SA], seq 67195418, ack 99659207, win 65160, options [mss 1460,ts 1603279370 990385264,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.52555: flags [SA], seq 3860588226, ack 4198299131, win 65160, options [mss 1460,ts 1603279370 990385264,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.52556: flags [SA], seq 618064228, ack 2660334177, win 65160, options [mss 1460,ts 1603279370 990385264,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.52557: flags [SA], seq 3620651171, ack 693903928, win 65160, options [mss 1460,ts 1603279370 990385264,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.52558: flags [SA], seq 4249802168, ack 1809403648, win 65160, options [mss 1460,ts 1603279370 990385264,ws 7], length 0
0.001 ( 0.001) IP macos.52559 > linux.8888: flags [S], seq 4212668517, win 65535, options [mss 1460,ws 6,ts 990385264 0], length 0
0.001 ( 0.000) IP macos.52560 > linux.8888: flags [S], seq 2104951382, win 65535, options [mss 1460,ws 6,ts 990385264 0], length 0
0.001 ( 0.000) IP macos.52551 > linux.8888: flags [A], seq 274874868, ack 254823053, win 2058, options [ts 990385264 1603279394], length 0
0.001 ( 0.000) IP macos.52552 > linux.8888: flags [A], seq 3641250080, ack 433081044, win 2058, options [ts 990385264 1603279394], length 0
0.001 ( 0.000) IP linux.8888 > macos.52559: flags [SA], seq 548771131, ack 4212668518, win 65160, options [mss 1460,ts 1603279370 990385264,ws 7], length 0
0.001 ( 0.000) IP linux.8888 > macos.52560: flags [SA], seq 3472646427, ack 2104951383, win 65160, options [mss 1460,ts 1603279370 990385264,ws 7], length 0
0.001 ( 0.000) IP macos.52553 > linux.8888: flags [A], seq 2153714275, ack 2399418258, win 2058, options [ts 990385264 1603279394], length 0
0.001 ( 0.000) IP macos.52554 > linux.8888: flags [A], seq 99659207, ack 67195419, win 2058, options [ts 990385264 1603279370], length 0
0.001 ( 0.000) IP macos.52555 > linux.8888: flags [A], seq 4198299131, ack 3860588227, win 2058, options [ts 990385264 1603279370], length 0
0.001 ( 0.000) IP macos.52556 > linux.8888: flags [A], seq 2660334177, ack 618064229, win 2058, options [ts 990385264 1603279370], length 0
0.001 ( 0.000) IP macos.52557 > linux.8888: flags [A], seq 693903928, ack 3620651172, win 2058, options [ts 990385264 1603279370], length 0
0.001 ( 0.000) IP macos.52558 > linux.8888: flags [A], seq 1809403648, ack 4249802169, win 2058, options [ts 990385264 1603279370], length 0
0.001 ( 0.000) IP macos.52559 > linux.8888: flags [A], seq 4212668518, ack 548771132, win 2058, options [ts 990385264 1603279370], length 0
0.001 ( 0.000) IP macos.52560 > linux.8888: flags [A], seq 2104951383, ack 3472646428, win 2058, options [ts 990385264 1603279370], length 0
从上面的输出看:
1. macos向Linux发送了10个SYN segments
2. linux向macos发送了10个SYN+ACK segments
3. macos向Linux发送了10个ACK segments
由此可知,10个连接全部完成了三次握手
在macos上查看连接
代码: 全选
macos $ netstat -f inet -p tcp -an | grep 8888 | sort
tcp4 0 0 192.168.0.3.52551 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.52552 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.52553 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.52554 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.52555 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.52556 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.52557 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.52558 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.52559 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.52560 192.168.0.6.8888 ESTABLISHED
从netstat的输出看,10个连接都处于ESTABLISHED状态
在Linux上查看连接
代码: 全选
linux $ netstat -4tan | grep 8888 | sort
tcp 0 0 192.168.0.6:8888 192.168.0.3:52551 ESTABLISHED
tcp 0 0 192.168.0.6:8888 192.168.0.3:52552 ESTABLISHED
tcp 0 0 192.168.0.6:8888 192.168.0.3:52553 ESTABLISHED
tcp 0 0 192.168.0.6:8888 192.168.0.3:52554 ESTABLISHED
tcp 4 0 0.0.0.0:8888 0.0.0.0:* LISTEN
从netstat的输出看,只有前4个连接处于ESTABLISHED状态
这是符合预期的,因为全连接队列的大小是4,其他6个连接还在半连接队列里
为什么会出现这个现象?
原因是:
1. 当收到SYN时,如果全连接队列和半连接队列都没满,则响应SYN+ACK
2. 当收到ACK时,如果全连接队列没满,则完成三次握手,然后将完成三次握手的连接从半连接队列移动到全连接队列,此时如果全连接队列已经满了,则继续留在半连接队列
3. 当收到ACK时,如果全连接队列满了,则直接丢弃ACK,随后会重传SYN+ACK给客户端(这是默认行为,可以通过/proc/sys/net/ipv4/tcp_abort_on_overflow文件修改)
使能tcp_abort_on_overflow
文件/proc/sys/net/ipv4/tcp_abort_on_overflow是一个布尔值,默认是0,如果设为1,则:
当服务端收到ACK时,如果全连接队列满了,则发送RST segment给客户端
实验:
先使能tcp_abort_on_overflow
代码: 全选
linux $ sudo bash -c 'echo 1 > /proc/sys/net/ipv4/tcp_abort_on_overflow'
在linux上启动simpletcpserver
代码: 全选
linux $ ./simpletcpserver -d 100000 -q 3 8888 # -q 3表示指定backlog为3,那么全连接队列大小则为4,-d 100000表示延迟100000秒再调用accept
在macos os上启动backlogtest
代码: 全选
macos $ ./backlogtest -c 10 linux 8888 # 启动10个线程,同时发起连接
recv_timeout: Connection reset by peer
recv_timeout: Connection reset by peer
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
recv_timeout: Operation timed out
上面的程序每次执行有一定的随机性,每次执行输出不一样
有几点说明:
1. 对于connect函数,只要收到对端发送的SYN+ACK就返回成功
2. connect返回成功,不代表完成了三次握手,因为服务端可能忽略客户端发送的ACK,然后发送一个RST segment给客户端,此时客户端只能通过recv函数来检测这个错误,上述输出的前两个就是这种情况
3. 还有一种可能是服务端收到SYN segment时,全连接队列或半连接队列已经满了,这时,服务端会丢弃SYN,什么都不响应,在客户端的表现是connect返回ETIMEDOUT
在Linux上使用mytcpdump观察segments
代码: 全选
linux $ sudo ./mytcpdump port 8888
0.000 ( 0.000) IP macos.55756 > linux.8888: flags [S], seq 2861565099, win 65535, options [mss 1460,ws 6,ts 995360867 0], length 0
0.000 ( 0.000) IP linux.8888 > macos.55756: flags [SA], seq 3413470649, ack 2861565100, win 65160, options [mss 1460,ts 1608300459 995360867,ws 7], length 0
0.000 ( 0.000) IP macos.55757 > linux.8888: flags [S], seq 3088571366, win 65535, options [mss 1460,ws 6,ts 995360867 0], length 0
0.000 ( 0.000) IP macos.55758 > linux.8888: flags [S], seq 3910276556, win 65535, options [mss 1460,ws 6,ts 995360867 0], length 0
0.000 ( 0.000) IP macos.55756 > linux.8888: flags [A], seq 2861565100, ack 3413470650, win 2058, options [ts 995360867 1608300459], length 0
0.000 ( 0.000) IP linux.8888 > macos.55757: flags [SA], seq 1016507251, ack 3088571367, win 65160, options [mss 1460,ts 1608300459 995360867,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.55758: flags [SA], seq 1110096341, ack 3910276557, win 65160, options [mss 1460,ts 1608300459 995360867,ws 7], length 0
0.000 ( 0.000) IP macos.55759 > linux.8888: flags [S], seq 3792309566, win 65535, options [mss 1460,ws 6,ts 995360867 0], length 0
0.000 ( 0.000) IP macos.55760 > linux.8888: flags [S], seq 1721799658, win 65535, options [mss 1460,ws 6,ts 995360867 0], length 0
0.000 ( 0.000) IP macos.55761 > linux.8888: flags [S], seq 4062300956, win 65535, options [mss 1460,ws 6,ts 995360867 0], length 0
0.000 ( 0.000) IP macos.55762 > linux.8888: flags [S], seq 431299088, win 65535, options [mss 1460,ws 6,ts 995360867 0], length 0
0.000 ( 0.000) IP macos.55763 > linux.8888: flags [S], seq 842395789, win 65535, options [mss 1460,ws 6,ts 995360867 0], length 0
0.000 ( 0.000) IP macos.55757 > linux.8888: flags [A], seq 3088571367, ack 1016507252, win 2058, options [ts 995360867 1608300459], length 0
0.000 ( 0.000) IP macos.55758 > linux.8888: flags [A], seq 3910276557, ack 1110096342, win 2058, options [ts 995360867 1608300459], length 0
0.000 ( 0.000) IP macos.55764 > linux.8888: flags [S], seq 3893580623, win 65535, options [mss 1460,ws 6,ts 995360867 0], length 0
0.000 ( 0.000) IP linux.8888 > macos.55759: flags [SA], seq 3977851215, ack 3792309567, win 65160, options [mss 1460,ts 1608300459 995360867,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.55760: flags [SA], seq 3607323909, ack 1721799659, win 65160, options [mss 1460,ts 1608300426 995360867,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.55761: flags [SA], seq 4253133500, ack 4062300957, win 65160, options [mss 1460,ts 1608300426 995360867,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.55762: flags [SA], seq 139184709, ack 431299089, win 65160, options [mss 1460,ts 1608300426 995360867,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.55763: flags [SA], seq 1964370174, ack 842395790, win 65160, options [mss 1460,ts 1608300426 995360867,ws 7], length 0
0.000 ( 0.000) IP linux.8888 > macos.55764: flags [SA], seq 2372681031, ack 3893580624, win 65160, options [mss 1460,ts 1608300459 995360867,ws 7], length 0
0.000 ( 0.000) IP macos.55765 > linux.8888: flags [S], seq 2037093638, win 65535, options [mss 1460,ws 6,ts 995360867 0], length 0
0.000 ( 0.000) IP macos.55759 > linux.8888: flags [A], seq 3792309567, ack 3977851216, win 2058, options [ts 995360867 1608300459], length 0
0.000 ( 0.000) IP macos.55760 > linux.8888: flags [A], seq 1721799659, ack 3607323910, win 2058, options [ts 995360867 1608300426], length 0
0.000 ( 0.000) IP macos.55761 > linux.8888: flags [A], seq 4062300957, ack 4253133501, win 2058, options [ts 995360867 1608300426], length 0
0.000 ( 0.000) IP macos.55762 > linux.8888: flags [A], seq 431299089, ack 139184710, win 2058, options [ts 995360867 1608300426], length 0
0.000 ( 0.000) IP macos.55763 > linux.8888: flags [A], seq 842395790, ack 1964370175, win 2058, options [ts 995360868 1608300426], length 0
0.001 ( 0.001) IP linux.8888 > macos.55765: flags [SA], seq 2492765422, ack 2037093639, win 65160, options [mss 1460,ts 1608300460 995360867,ws 7], length 0
0.001 ( 0.000) IP macos.55764 > linux.8888: flags [A], seq 3893580624, ack 2372681032, win 2058, options [ts 995360868 1608300459], length 0
0.001 ( 0.000) IP macos.55765 > linux.8888: flags [A], seq 2037093639, ack 2492765423, win 2058, options [ts 995360868 1608300460], length 0
0.001 ( 0.000) IP linux.8888 > macos.55764: flags [R], seq 2372681032, win 0, options [], length 0
0.001 ( 0.000) IP linux.8888 > macos.55765: flags [R], seq 2492765423, win 0, options [], length 0
从上面的输出看:
有8个连接完成了三次握手
有2个连接被reset了
在macos上查看连接
代码: 全选
macos $ netstat -f inet -p tcp -an | grep 8888 | sort
tcp4 0 0 192.168.0.3.55756 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.55757 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.55758 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.55759 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.55760 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.55761 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.55762 192.168.0.6.8888 ESTABLISHED
tcp4 0 0 192.168.0.3.55763 192.168.0.6.8888 ESTABLISHED
在Linux上查看连接
代码: 全选
linux $ netstat -4tan | grep 8888 | sort
tcp 0 0 192.168.0.6:8888 192.168.0.3:55756 ESTABLISHED
tcp 0 0 192.168.0.6:8888 192.168.0.3:55757 ESTABLISHED
tcp 0 0 192.168.0.6:8888 192.168.0.3:55758 ESTABLISHED
tcp 0 0 192.168.0.6:8888 192.168.0.3:55759 ESTABLISHED
tcp 4 0 0.0.0.0:8888 0.0.0.0:* LISTEN
从上面的输出看:
有4个连接被放到了全连接队列里,还有4个完成了三次握手的连接在半连接队列里
总结: