Kubernetes容器编排技术网络流量分析

发布时间：2019-06-18 16:24

杜平

（中国联通晋中分公司 030600）

摘要

容器技术近年来在各大企业、互联网公司作为最先进的云虚拟化技术发挥了重要的作用，为业务快速迭代和发展提供了有效支持。本文深度分析了Kubernetes容器编排技术在Canal网络方案的不同场景中的数据流向、原理、故障排查方法等。

关键词：

Docker Kubernetes Canal容器 overlay Deployment

ABSTRACT

In recent years, some large enterprise start to adopt container technology as virtualization to serve business grow fast. This thesis analyzes container network technology by Kubernetes and Canal base on experiment environment for data flow, route and troubleshooting methodology.

KEY WORDS:

Docker Kubernetes Canal overlay Deployment

1 概述

容器云技术已经越来越被企业所采用，助力传统企业业务的数字化转型，应对互联网大潮下的业务冲击。Kubernetes作为容器编排的事实标准已被49%的企业所采用，而另外的企业也在考虑和测试Kubernetes在业务环境里使用的可能性。联通内部互联网类应用越来越多，未来可以使用容器技术提升业务的迭代速度和效率，更好地为客户服务。容器技术中网络是比较复杂的组成部分，本文深度剖析了Kubernetes与Canal构建的容器云网络的数据流向和相关故障排查方法。

2 研究环境

此次研究课题所使用的环境如下图1所示：

图1：研究环境

环境说明：

1. 客户端：1台

2. 服务器：2台，Centos7-1：172.16.240.131，Centos7-2：172.16.240.132

3. 服务器操作系统：Centos7.3

4. 服务器安装的平台服务：Kubernetes（包括：apiserver，scheduler，controller，etcd，kubelet，kubeproxy），Docker，Canal网络组件等。

5. 测试所使用的应用：service：ghost，IP地址：10.250.0.40，应用暴露端口：31266

6. 应用后端的pod的IP地址：10.244.1.5，10.244.0.74

7. 客户端IP：172.16.240.1

3 网络流量分析

3.1 容器平台内不同节点上应用pod之间访问（ghost-ep1访问ghost-ep2）

###centos7-1###查看ghost的2个副本

[root@centos7-1 ~]# kubectl get pods -o wide | grep ghost

NAME READY STATUS RESTARTS AGE IP NODE

ghost-55669748d5-txmpg 1/1 Running 0 1m 10.244.1.5 centos7-1

ghost-55669748d5-pgnhw 1/1 Running 1 7d 10.244.0.74 centos7-2

###centos7-1###进入pod内部：ghost-55669748d5-txmpg

[root@centos7-1 ~]# kubectl exec -ti ghost-55669748d5-txmpg bash

###pod：ghost-55669748d5-txmpg###查看ghost-ep1的ip地址是10.244.1.5/32，pod内的网卡eth0

root@ghost-55669748d5-txmpg:/var/lib/ghost# ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

3: eth0@if8: mtu 1500 qdisc noqueue state UP group default

link/ether 3e:0c:94:46:7b:98 brd ff:ff:ff:ff:ff:ff

inet 10.244.1.5/32 scope global eth0

valid_lft forever preferred_lft forever

###pod：ghost-55669748d5-txmpg###查看pod内的路由，缺省路由的下一跳是169.254.1.1，通过设备eth0出去

root@ghost-55669748d5-txmpg:/var/lib/ghost# ip route

default via 169.254.1.1 dev eth0

169.254.1.1 dev eth0 scope link

###pod：ghost-55669748d5-txmpg###查看pod内的网络邻居，发现169.254.1.1的mac地址是 56:aa:dd:0b:e9:03

root@ghost-55669748d5-txmpg:/var/lib/ghost# ip nei

172.16.240.131 dev eth0 lladdr 56:aa:dd:0b:e9:03 STALE

169.254.1.1 dev eth0 lladdr 56:aa:dd:0b:e9:03 STALE

###centos7-1###查看宿主机所有接口，56:aa:dd:0b:e9:03在设备calif0e9df43659上

[root@centos7-1 ~]# ip addr show

1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: ens33: mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:b2:8d:83 brd ff:ff:ff:ff:ff:ff

inet 172.16.240.131/24 brd 172.16.240.255 scope global ens33

valid_lft forever preferred_lft forever

inet6 fe80::c100:4631:34fa:3256/64 scope link

valid_lft forever preferred_lft forever

3: docker0: mtu 1500 qdisc noqueue state DOWN

link/ether 02:42:fc:fd:bd:65 brd ff:ff:ff:ff:ff:ff

inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

valid_lft forever preferred_lft forever

4: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN

link/ether 8a:40:90:1c:cf:7e brd ff:ff:ff:ff:ff:ff

inet 10.244.1.0/32 scope global flannel.1

valid_lft forever preferred_lft forever

inet6 fe80::8840:90ff:fe1c:cf7e/64 scope link

valid_lft forever preferred_lft forever

8: calif0e9df43659@if3: mtu 1500 qdisc noqueue state UP

link/ether 56:aa:dd:0b:e9:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0

inet6 fe80::54aa:ddff:fe0b:e903/64 scope link

valid_lft forever preferred_lft forever

###centos7-1###查看设备calif0e9df43659，proxy_arp=1，做了arp代答（代替pod应答arp请求）

[root@centos7-1 ~]# sysctl -a | grep calif0e9df43659 | grep proxy_arp

sysctl: reading key 'net.ipv6.conf.all.stable_secret'

sysctl: reading key 'net.ipv6.conf.calif0e9df43659.stable_secret'

sysctl: reading key 'net.ipv6.conf.default.stable_secret'

sysctl: reading key 'net.ipv6.conf.docker0.stable_secret'

sysctl: reading key 'net.ipv6.conf.ens33.stable_secret'

sysctl: reading key 'net.ipv6.conf.flannel/1.stable_secret'

sysctl: reading key 'net.ipv6.conf.lo.stable_secret'

net.ipv4.conf.calif0e9df43659.proxy_arp = 1

net.ipv4.conf.calif0e9df43659.proxy_arp_pvlan = 0

###centos7-1###查看宿主机路由表，去往ghost-ep2的路由指示从设备flannel.1发出

[root@centos7-1 ~]# ip route

default via 172.16.240.2 dev ens33 proto static metric 100

10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink

10.244.1.5 dev calif0e9df43659 scope link

172.16.240.0/24 dev ens33 proto kernel scope link src 172.16.240.131 metric 100

172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1

[root@centos7-1 ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 172.16.240.2 0.0.0.0 UG 100 0 0 ens33

10.244.0.0 10.244.0.0 255.255.255.0 UG 0 0 0 flannel.1

10.244.1.5 0.0.0.0 255.255.255.255 UH 0 0 0 calif0e9df43659

172.16.240.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33

172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0

###centos7-1###查看设备flannel.1，vxlan的设备，vni=1，本机地址是172.16.240.131，从设备ens33发出

[root@centos7-1 ~]# ip -d link show flannel.1

4: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT

link/ether 8a:40:90:1c:cf:7e brd ff:ff:ff:ff:ff:ff promiscuity 0

vxlan id 1 local 172.16.240.131 dev ens33 srcport 0 0 dstport 8472 nolearning ageing 300 addrgenmode eui64

###centos7-1###查看宿主机的网络邻居，去往10.244.0.0的数据包，从设备 flannel.1出，邻居的mac是32:a9:79:6d:0b:b1

[root@centos7-1 ~]# ip nei

172.16.240.132 dev ens33 lladdr 00:0c:29:b5:b6:92 REACHABLE

10.244.0.0 dev flannel.1 lladdr 32:a9:79:6d:0b:b1 PERMANENT

172.16.240.1 dev ens33 lladdr 00:50:56:c0:00:08 DELAY

10.244.1.4 dev calif7ade911ae8 lladdr 12:9c:95:cd:4e:cd STALE

172.16.240.2 dev ens33 lladdr 00:50:56:fc:f9:04 DELAY

###centos7-1###查看vxlan的fdb表，发现对端vtep的ip是172.16.240.132

FDB表是二层转发表，

[root@centos7-1 ~]# bridge fdb

01:00:5e:00:00:01 dev ens33 self permanent

33:33:00:00:00:01 dev ens33 self permanent

33:33:ff:fa:32:56 dev ens33 self permanent

33:33:00:00:00:01 dev docker0 self permanent

01:00:5e:00:00:01 dev docker0 self permanent

02:42:fc:fd:bd:65 dev docker0 vlan 1 master docker0 permanent

02:42:fc:fd:bd:65 dev docker0 master docker0 permanent

32:a9:79:6d:0b:b1 dev flannel.1 dst 172.16.240.132 self permanent

33:33:00:00:00:01 dev calif0e9df43659 self permanent

01:00:5e:00:00:01 dev calif0e9df43659 self permanent

33:33:ff:0b:e9:03 dev calif0e9df43659 self permanent

###centos7-2###TCP/IP是内核态行为，拆掉数据包，发现是vxlan，本地有路由，设备是cali4a6ea607e95（代答pod的arp）

[root@centos7-2 ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 172.16.240.2 0.0.0.0 UG 100 0 0 ens33

10.244.0.65 0.0.0.0 255.255.255.255 UH 0 0 0 cali185c333c9d5

10.244.0.69 0.0.0.0 255.255.255.255 UH 0 0 0 cali4ea182e3288

10.244.0.72 0.0.0.0 255.255.255.255 UH 0 0 0 cali807b30de140

10.244.0.74 0.0.0.0 255.255.255.255 UH 0 0 0 cali4a6ea607e95

10.244.0.82 0.0.0.0 255.255.255.255 UH 0 0 0 cali1a3e756d696

10.244.0.83 0.0.0.0 255.255.255.255 UH 0 0 0 calic440f455693

10.244.0.84 0.0.0.0 255.255.255.255 UH 0 0 0 cali0dd295201d1

10.244.0.85 0.0.0.0 255.255.255.255 UH 0 0 0 cali1c1c361bfa6

10.244.0.86 0.0.0.0 255.255.255.255 UH 0 0 0 caliccec0fc857b

10.244.0.88 0.0.0.0 255.255.255.255 UH 0 0 0 cali02a732f7d51

10.244.1.0 10.244.1.0 255.255.255.0 UG 0 0 0 flannel.1

172.16.240.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33

172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0

###centos7-2###cali4a6ea607e95转给pod：ghost-55669748d5-pgnhw

3.2 容器平台外部客户端访问容器平台内部的应用pod（client访问ghost-ep2）

###centos7-1###查看ghost的service，port是31266

[root@centos7-1 ~]# kubectl get svc ghost -o wide

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR

ghost NodePort 10.250.0.40 2368:31266/TCP 10d run=ghost

###centos7-1###查看ghost的endpoints，共2个ep

[root@centos7-1 ~]# kubectl get endpoints ghost -o wide

NAME ENDPOINTS AGE

ghost 10.244.0.74:2368,10.244.1.5:2368 10d

###centos7-1###查看pod信息，pod在centos7-2上

[root@centos7-1 ~]# kubectl get pod ghost-55669748d5-pgnhw -o wide

NAME READY STATUS RESTARTS AGE IP NODE

ghost-55669748d5-pgnhw 1/1 Running 1 7d 10.244.0.74 centos7-2

去程发送端PREROUTING: 请求报文 172.16.240.1:任意端口 ----> 172.16.240.131:31266

###centos7-1###查看PREROUTING的chain的nat表中内容

[root@centos7-1 ~]# iptables -vnL PREROUTING -t nat

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

1749 142K cali-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:6gwbT8clXdHdC1b1 */

1749 142K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */

1188 74424 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL

###centos7-1###查看KUBE-SERVICE chain的nat表中的内容，优先匹配Cluster IP的报文，最后默认匹配KUBE-NODEPORTS

[root@centos7-1 ~]# iptables -vnL KUBE-SERVICES -t nat

Chain KUBE-SERVICES (2 references)

pkts bytes target prot opt in out source destination

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.101 /* default/nginx: cluster IP */ tcp dpt:80

0 0 KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- * * 0.0.0.0/0 10.250.0.101 /* default/nginx: cluster IP */ tcp dpt:80

0 0 KUBE-MARK-MASQ udp -- * * !10.244.0.0/16 10.250.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53

0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.250.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53

0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.250.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.151 /* default/default-http-backend: cluster IP */ tcp dpt:80

0 0 KUBE-SVC-WMDDEEP2PS5RXKUQ tcp -- * * 0.0.0.0/0 10.250.0.151 /* default/default-http-backend: cluster IP */ tcp dpt:80

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.40 /* default/ghost: cluster IP */ tcp dpt:2368

0 0 KUBE-SVC-TFV3VPHN2LAGKCVY tcp -- * * 0.0.0.0/0 10.250.0.40 /* default/ghost: cluster IP */ tcp dpt:2368

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.11 /* default/ghost-ingress: cluster IP */ tcp dpt:2368

0 0 KUBE-SVC-BXCGFFQYYR74AICZ tcp -- * * 0.0.0.0/0 10.250.0.11 /* default/ghost-ingress: cluster IP */ tcp dpt:2368

0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.250.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443

0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.250.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443

4 240 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

###centos7-1###查看KUBE-NODEPORTS chain的nat表中的内容，先打MASQMARK，之后匹配规则

[root@centos7-1 ~]# iptables -vnL KUBE-NODEPORTS -t nat

Chain KUBE-NODEPORTS (1 references)

pkts bytes target prot opt in out source destination

0 0 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx: */ tcp dpt:30044

0 0 KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx: */ tcp dpt:30044

0 0 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/ghost: */ tcp dpt:31266

0 0 KUBE-SVC-TFV3VPHN2LAGKCVY tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/ghost: */ tcp dpt:31266

###centos7-1###查看KUBE-MARK-MASQ chain的nat表的内容，数据包返回时要做SNAT，对DNAT的报文打0x4000 PKT mark

[root@centos7-1 ~]# iptables -vnL KUBE-MARK-MASQ -t nat

Chain KUBE-MARK-MASQ (18 references)

pkts bytes target prot opt in out source destination

0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000

###centos7-1###查看KUBE-SVC-TFV3VPHN2LAGKCVY chain的nat表中的内容，有2个endpoints，平均分发

[root@centos7-1 ~]# iptables -vnL KUBE-SVC-TFV3VPHN2LAGKCVY -t nat

Chain KUBE-SVC-TFV3VPHN2LAGKCVY (2 references)

pkts bytes target prot opt in out source destination

0 0 KUBE-SEP-UJXTJ7PNRUN6D2JD all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/ghost: */ statistic mode random probability 0.50000000000

0 0 KUBE-SEP-MNOUPYFQRWSOE45J all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/ghost: */

###centos7-1###查看KUBE-SEP-UJXTJ7PNRUN6D2JD chain的nat表中的内容，做DNAT，将目的ip和port（172.16.240.131:30266）转换成对应容器的ip和port（10.244.0.74:2368）

[root@centos7-1 ~]# iptables -vnL KUBE-SEP-UJXTJ7PNRUN6D2JD -t nat

Chain KUBE-SEP-UJXTJ7PNRUN6D2JD (1 references)

pkts bytes target prot opt in out source destination

0 0 KUBE-MARK-MASQ all -- * * 10.244.0.74 0.0.0.0/0 /* default/ghost: */

0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/ghost: */ tcp to:10.244.0.74:2368

去程发送端FORWARD: 路由查询出的设备：请求报文 172.16.240.1:任意端口 ----> 10.244.0.74:2368

###centos7-1###查看宿主机路由表，找到10.240.0的路由，需要走到flannel.1接口上

[root@centos7-1 ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 172.16.240.2 0.0.0.0 UG 100 0 0 ens33

10.244.0.0 10.244.0.0 255.255.255.0 UG 0 0 0 flannel.1

10.244.1.5 0.0.0.0 255.255.255.255 UH 0 0 0 calif0e9df43659

172.16.240.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33

172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0

###centos7-1###查看接口设备，vxlan的ep，本端地址：172.16.240.131

[root@centos7-1 ~]# ip -d link show flannel.1

4: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT

link/ether 8a:40:90:1c:cf:7e brd ff:ff:ff:ff:ff:ff promiscuity 0

vxlan id 1 local 172.16.240.131 dev ens33 srcport 0 0 dstport 8472 nolearning ageing 300 addrgenmode eui64

去程发送端POSTROUTING:

###centos7-1###查看POSTROUTING的chain的nat表内容

[root@centos7-1 ~]# iptables -vnL POSTROUTING -t nat

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

60312 3695K cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:O3lYWMrLQYEMJtB5 */

0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0

61788 3789K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */

35 2716 RETURN all -- * * 10.244.0.0/16 10.244.0.0/16

6 360 MASQUERADE all -- * * 10.244.0.0/16 !224.0.0.0/4

0 0 RETURN all -- * * !10.244.0.0/16 10.244.1.0/24

0 0 MASQUERADE all -- * * !10.244.0.0/16 10.244.0.0/16

###centos7-1###查看KUBE-POSTROUTING的chain的nat表内容

[root@centos7-1 ~]# iptables -vnL KUBE-POSTROUTING -t nat

Chain KUBE-POSTROUTING (1 references)

pkts bytes target prot opt in out source destination

0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

出接口为flannel.1，MASQUERADE通过inet_select_addr（选择flannel.1网卡的ip作为原ip）从flannel.1上先把IP作SNAT，此时报文请求变为10.244.1.0:任意端口 ----> 10.244.0.74:2368

去程发送端Vxlan通信：10.244.1.0:任意端口 ----> 10.244.0.74:2368

从vxlan端口(flannel.1)发送报文，驱动调用vxlan_xmit，通过目的mac查询vxlan_fdb先把remote ip（外层目的ip），即172.16.240.132，从vxlan_dev中获取源local ip 172.16.240.131（&vxlan->cfg.saddr），然后调iptunnel_xmit->ip_local_out从对应物理网口将报文送出去。

###centos7-1###查看宿主机的网络邻居，去往10.244.0.0的数据包，从设备 flannel.1出，邻居的mac是32:a9:79:6d:0b:b1

[root@centos7-1 ~]# ip nei

172.16.240.132 dev ens33 lladdr 00:0c:29:b5:b6:92 REACHABLE

10.244.0.0 dev flannel.1 lladdr 32:a9:79:6d:0b:b1 PERMANENT

172.16.240.1 dev ens33 lladdr 00:50:56:c0:00:08 DELAY

10.244.1.4 dev calif7ade911ae8 lladdr 12:9c:95:cd:4e:cd STALE

172.16.240.2 dev ens33 lladdr 00:50:56:fc:f9:04 DELAY

###centos7-1###查看vxlan的fdb表，发现对端vtep的ip是172.16.240.132

FDB表是二层转发表，

[root@centos7-1 ~]# bridge fdb

01:00:5e:00:00:01 dev ens33 self permanent

33:33:00:00:00:01 dev ens33 self permanent

33:33:ff:fa:32:56 dev ens33 self permanent

33:33:00:00:00:01 dev docker0 self permanent

01:00:5e:00:00:01 dev docker0 self permanent

02:42:fc:fd:bd:65 dev docker0 vlan 1 master docker0 permanent

02:42:fc:fd:bd:65 dev docker0 master docker0 permanent

32:a9:79:6d:0b:b1 dev flannel.1 dst 172.16.240.132 self permanent

33:33:00:00:00:01 dev calif0e9df43659 self permanent

01:00:5e:00:00:01 dev calif0e9df43659 self permanent

33:33:ff:0b:e9:03 dev calif0e9df43659 self permanent

###centos7-1###查看POSTROUTING的chain的nat表内容，由于 POSTROUTING 有如下匹配规则，所以 vxlan 报文不会再做一次 SNAT

[root@centos7-1 ~]# iptables -vnL POSTROUTING -t nat

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

60312 3695K cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:O3lYWMrLQYEMJtB5 */

0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0

61788 3789K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */

35 2716 RETURN all -- * * 10.244.0.0/16 10.244.0.0/16

6 360 MASQUERADE all -- * * 10.244.0.0/16 !224.0.0.0/4

0 0 RETURN all -- * * !10.244.0.0/16 10.244.1.0/24

0 0 MASQUERADE all -- * * !10.244.0.0/16 10.244.0.0/16

去程接收端：

Vxlan udp报文处理经过vxlan_rcv (ens3) -> netif_rx(vxlandev skb)

其中vxlan_rcv中会使用vxlan_vs_find_vni(vs, vxlan_vni(vxlan_hdr(skb)->vx_vni))通过vxlan报文vni来查找对应的vxlan_dev，所以vxlan网口vni要一致。

###centos7-2###查看设备flannel.1，vxlan的设备，vni=1，本机地址是172.16.240.132，从设备ens33发出

[root@centos7-2 ~]# ip -d link show flannel.1

13: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT

link/ether 32:a9:79:6d:0b:b1 brd ff:ff:ff:ff:ff:ff promiscuity 0

vxlan id 1 local 172.16.240.132 dev ens33 srcport 0 0 dstport 8472 nolearning ageing 300 addrgenmode eui64