External traffic policy
Services can be configured to have externalTrafficPolicy Cluster (default) or Local. If the externalTrafficPolicy is Cluster (default) the ECFE speaker on a given node will announce the service External IP no matter if the node has a service endpoint. If the externalTrafficPolicy is Local the ECFE speaker on a given node will announce the service External IP, only if a healthy service endpoint exist on the node.

From https://adp.ericsson.se/marketplace/external-connectivity-frontend/documentation/development/dpi/application-developers-guide

Ingress controller service manifest.
eccd@control-plane-wfb88:~> kubectl get svc ingress-nginx -n ingress-nginx -o yaml
apiVersion: v1
kind: Service
metadata:
{…}
spec:
clusterIP: 10.97.74.168
externalTrafficPolicy: Cluster
ports:

  • name: http
    nodePort: 32151
    port: 80
    protocol: TCP
    targetPort: 80
  • name: https
    nodePort: 31255
    port: 443
    protocol: TCP
    targetPort: 443
    selector:
    app: ingress-nginx
    sessionAffinity: None
    type: LoadBalancer
    status:
    loadBalancer:
    ingress:
    • ip: 10.33.151.160

let’s use dc173prometheus1.cloud.k2.ericsson.se as example.

Source IP: seroius01242
10.210.150.87/28
Dest: dc173prometheus1.cloud.k2.ericsson.se

  1. 1. Lookup DNS record:
  2. eccd@control-plane-wfb88:~> nslookup dc173prometheus1.cloud.k2.ericsson.se
  3. Server:         10.96.0.10
  4. Address:        10.96.0.10#53
  6. Non-authoritative answer:
  7. dc173prometheus1.cloud.k2.ericsson.se   canonical name = dc173dashboard1.cloud.k2.ericsson.se.
  8. Name:   dc173dashboard1.cloud.k2.ericsson.se
  9. Address: 10.33.151.160
  11. eccd@control-plane-wfb88:~>
  13. 2. Verify FQDN is exposed via Kubernetes ingress
  14. eccd@control-plane-wfb88:~> kubectl get ing -A
  15. NAMESPACE     NAME                                  CLASS    HOSTS                                               ADDRESS                                           PORTS     AGE
  16. monitoring    prometheus-server                     <none>   dc173prometheus1.cloud.k2.ericsson.se               10.0.10.102,10.0.10.103,10.0.10.104,10.0.10.108   80        28d
  18. Expose Prometheus via ingress if not ready done:
  19. cat >> PM-ingress.yaml << EOF
  20. apiVersion: extensions/v1beta1
  21. kind: Ingress
  22. metadata:
  23. labels:
  24. app: eric-pm-server
  25. component: server
  26. name: prometheus-server
  27. namespace: monitoring
  28. spec:
  29. rules:
  30. - host: dc173prometheus1.cloud.k2.ericsson.se
  31. http:
  32. paths:
  33. - backend:
  34. serviceName: eric-pm-server
  35. servicePort: 9090
  36. path: /
  37. EOF
  39. Apply ingress rule
  40. kubectl apply -f PM-ingress.yaml 
  42. 3. Verify external IP is exposed by ingress-nginx service with type LoadBalancer.
  43. eccd@control-plane-wfb88:~> kubectl get svc -A
  44. NAMESPACE       NAME                                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                 AGE
  45. ingress-nginx   ingress-nginx                          LoadBalancer   10.97.74.168     10.33.151.160   80:32151/TCP,443:31255/TCP              28d
  47. At this stage, DNS record is consistent with Kubernetes ingress configuration.
  50. 4. Check ECFE configmap
  51. eccd@control-plane-wfb88:~> kubectl get cm ecfe-ccdadm -n kube-system  -o yaml
  52. apiVersion: v1
  53. data:
  54. config: |
  55. bgp-bfd-peers:
  56. - peer-address: 10.33.151.66
  57. peer-asn: 4200000000
  58. my-asn: 4259841731
  59. hold-time: 3s
  60. min-rx: 300ms
  61. min-tx: 300ms
  62. multiplier: 3
  63. my-address-pools:
  64. - ingress
  65. - oam-pool
  66. - peer-address: 10.33.151.67
  67. peer-asn: 4200000000
  68. my-asn: 4259841731
  69. hold-time: 3s
  70. min-rx: 300ms
  71. min-tx: 300ms
  72. multiplier: 3
  73. my-address-pools:
  74. - ingress
  75. - oam-pool
  76. - peer-address: 10.33.152.66
  77. peer-asn: 4200000000
  78. my-asn: 4259841731
  79. hold-time: 3s
  80. min-rx: 300ms
  81. min-tx: 300ms
  82. multiplier: 3
  83. my-address-pools:
  84. - traffic-pool
  85. - peer-address: 10.33.152.67
  86. peer-asn: 4200000000
  87. my-asn: 4259841731
  88. hold-time: 3s
  89. min-rx: 300ms
  90. min-tx: 300ms
  91. multiplier: 3
  92. my-address-pools:
  93. - traffic-pool
  94. - peer-address: 172.80.11.2
  95. peer-asn: 4200000000
  96. my-asn: 4259841731
  97. hold-time: 3s
  98. min-rx: 300ms
  99. min-tx: 300ms
  100. multiplier: 3
  101. my-address-pools:
  102. - traffic2-pool
  103. - peer-address: 172.80.11.3
  104. peer-asn: 4200000000
  105. my-asn: 4259841731
  106. hold-time: 3s
  107. min-rx: 300ms
  108. min-tx: 300ms
  109. multiplier: 3
  110. my-address-pools:
  111. - traffic2-pool
  112. address-pools:
  113. - name: ingress
  114. protocol: bgp
  115. addresses:
  116. - 10.33.151.160/32
  117. - name: oam-pool
  118. protocol: bgp
  119. addresses:
  120. - 10.33.151.161-10.33.151.167
  121. auto-assign: false
  122. - name: traffic-pool
  123. protocol: bgp
  124. addresses:
  125. - 10.33.152.16/28
  126. auto-assign: false
  127. - name: traffic2-pool
  128. protocol: bgp
  129. addresses:
  130. - 10.33.152.32/28
  131. auto-assign: false
  132. kind: ConfigMap
  134. VIP 10.33.151.160/32 is configured in ECFE ingress address pool. It's announced to BGP peer 10.33.151.66 and 10.33.151.67.
  136. 5. Verify SLX configuration
  137. DC173-SLX-L1A# show run int ve
  138. interface Ve 805
  139. vrf forwarding DC173_CCD1_om_vr
  140. ip anycast-address 10.33.151.65/27
  141. ip address 10.33.151.66/27
  142. no shutdown
  143. !
  144. DC173-SLX-L1B# show run int ve
  145. interface Ve 805
  146. vrf forwarding DC173_CCD1_om_vr
  147. ip anycast-address 10.33.151.65/27
  148. ip address 10.33.151.67/27
  149. no shutdown
  150. !
  151. DC173-SLX-L1B# 
  154. DC173-SLX-L1A# show ip bgp vrf DC173_CCD1_om_vr
  155. Total number of BGP Routes: 31
  156. Status codes: s suppressed, d damped, h history, * valid, > best, i internal, S stale, x best-external
  157. Origin codes: i - IGP, e - EGP, ? - incomplete
  158. Network            Next Hop        RD         MED        LocPrf     Weight Path
  159. *>x 0.0.0.0/0          10.90.0.2                   none      100        0      100 i
  160. *   0.0.0.0/0          10.90.0.6                   none      100        0      100 i
  161. *i  0.0.0.0/0          21.1.1.1                    none      100        0      100 i
  162. *>  10.33.151.64/27    0.0.0.0                    0          100        32768  ?
  163. *i  10.33.151.64/27    21.1.1.1                   0          100        0      ?
  164. *>  10.33.151.96/27    0.0.0.0                    0          100        32768  ?
  165. *i  10.33.151.96/27    21.1.1.1                   0          100        0      ?
  166. *>  10.33.151.128/27   0.0.0.0                    0          100        32768  ?
  167. *i  10.33.151.128/27   21.1.1.1                   0          100        0      ?
  168. *>x 10.33.151.160/32   10.33.151.68                none      100        0      4259841731 i
  169. *   10.33.151.160/32   10.33.151.69                none      100        0      4259841731 i
  170. *   10.33.151.160/32   10.33.151.70                none      100        0      4259841731 i
  171. *   10.33.151.160/32   10.33.151.71                none      100        0      4259841731 i
  172. *   10.33.151.160/32   10.33.151.72                none      100        0      4259841731 i
  173. *   10.33.151.160/32   10.33.151.73                none      100        0      4259841731 i
  174. *   10.33.151.160/32   10.33.151.74                none      100        0      4259841731 i
  175. *   10.33.151.160/32   10.33.151.75                none      100        0      4259841731 i
  176. *   10.33.151.160/32   10.33.151.76                none      100        0      4259841731 i
  177. *   10.33.151.160/32   10.33.151.77                none      100        0      4259841731 i
  178. *   10.33.151.160/32   10.33.151.78                none      100        0      4259841731 i
  179. *i  10.33.151.160/32   21.1.1.1                    none      100        0      4259841731 i
  180. *>  10.33.151.176/28   0.0.0.0                    0          100        32768  ?
  181. *i  10.33.151.176/28   21.1.1.1                   0          100        0      ?
  182. *>  10.90.0.0/30       0.0.0.0                    0          100        32768  ?
  183. *>  10.90.0.4/30       0.0.0.0                    0          100        32768  ?
  184. *>i 10.90.1.0/30       21.1.1.1                   0          100        0      ?
  185. *>i 10.90.1.4/30       21.1.1.1                   0          100        0      ?
  186. *>  21.1.1.0/31        0.0.0.0                    0          100        32768  ?
  187. *i  21.1.1.0/31        21.1.1.1                   0          100        0      ?
  188. *>  172.80.70.0/24     0.0.0.0                    0          100        32768  ?
  189. *i  172.80.70.0/24     21.1.1.1                   0          100        0      ?
  190. DC173-SLX-L1A# 
  192. Dst=10.33.151.160, there are a few next hops, from 10.33.151.68 to 10.33.151.78.
  194. Traffic will be load balanced by ECMP towards those 11 IPs.
  196. <--------First load balancing by  ECMP - Layer 4-------->
  197. SLX supports ECMP for L3 forwarding using a modulo N hash algorithm implemented in the forwarding ASICs
  198. Traffic is forwarded to a specific nexthop derived by hashing on the 5-tuple frame data (IPSA, IPDA, IPProto, L4 SP, L4 DP) and computing the specific index using a module N method when N = pathCount in the ECMP list of Nexthops
  200. 6. Verify who owns those BGP speaker IP:
  201. eccd@control-plane-wfb88:~> for i in `kubectl get no -o json | jq -r '.items[].status.addresses[] | select(.type=="InternalIP") | .address'` ; do { ssh $i -q hostname & ssh $i -q ip a show ccd_ecfe_om | grep "inet " | awk -F'[: ]+' '{ print $3 }'; } | paste  -s; done
  202. control-plane-4qjt9     10.33.151.77/27
  203. control-plane-vxt5p     10.33.151.68/27 10.33.151.132/32
  204. control-plane-wfb88     10.33.151.78/27
  205. pool1-5db74894bb-tf54f  10.33.151.69/27
  206. pool2-9976d6447-2gnmp   10.33.151.73/27
  207. pool2-9976d6447-5bhnl   10.33.151.74/27
  208. pool2-9976d6447-djhjx   10.33.151.75/27
  209. pool2-9976d6447-ms9hg   10.33.151.71/27
  210. 10.33.151.70/27 pool2-9976d6447-nd4xl
  211. pool2-9976d6447-r776t   10.33.151.72/27
  212. pool2-9976d6447-xmbcz   10.33.151.76/27
  214. 7. Verify src=10.210.150.87, dst=10.33.151.160 can be seen in one of the BGP speaker node. One BGP speaker node has 1/11 chance to receive the ingress traffic.
  216. eccd@control-plane-wfb88:~> sudo tcpdump -vvveni ccd_ecfe_om 
  218. Frame 1: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
  219. Ethernet II, Src: ExtremeN_d6:8e:bc (00:04:96:d6:8e:bc), Dst: ea:a3:88:4e:f3:00 (ea:a3:88:4e:f3:00)
  220. Internet Protocol Version 4, Src: 10.210.150.87, Dst: 10.33.151.160
  221. Transmission Control Protocol, Src Port: 56200, Dst Port: 80, Seq: 0, Len: 0
  224. Frame 4: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
  225. Ethernet II, Src: ExtremeN_d6:88:76 (00:04:96:d6:88:76), Dst: ea:a3:88:4e:f3:00 (ea:a3:88:4e:f3:00)
  226. Internet Protocol Version 4, Src: 10.210.150.87, Dst: 10.33.151.160
  227. Transmission Control Protocol, Src Port: 56202, Dst Port: 80, Seq: 0, Len: 0
  229. 8. Verify the IP is forwarded by SLX.
  230. 00:04:96:d6:8e:bc belongs to SLX1.
  231. 00:04:96:d6:88:76 belongs to SLX2.
  233. DC173-SLX-L1A# show interface | begin "Ve 805"
  234. Ve 805 is up, line protocol is up
  235. Address is 0004.96d6.8ebc, Current address is 0004.96d6.8ebc
  236. DC173-SLX-L1B# show interface | begin "Ve 805"
  237. Ve 805 is up, line protocol is up
  238. Address is 0004.96d6.8876, Current address is 0004.96d6.8876
  239. Interface index (ifindex) is 1207960357 (0x48000325)
  242. iptables approach - Very important concept - user defined chain 
  245. Due to externalTrafficPolicy: Cluster, all BGP speaker node shall be able to see the iptables rule:
  246. a. All traffic destined 10.33.151.160/32 with dest port 443 will enter user defined KUBE-FW-4E7KSV2ABIFJRAUZ chain.
  248. control-plane-mjnp4:/home/eccd # iptables -t nat -S|grep 10.33.151.160
  249. -A KUBE-SERVICES -d 10.33.151.160/32 -p tcp -m comment --comment "ingress-nginx/ingress-nginx:https loadbalancer IP" -m tcp --dport 443 -j KUBE-FW-4E7KSV2ABIFJRAUZ <-next step b
  250. -A KUBE-SERVICES -d 10.33.151.160/32 -p tcp -m comment --comment "ingress-nginx/ingress-nginx:http loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-REQ4FPVT7WYF4VLA
  251. control-plane-mjnp4:/home/eccd # 
  253. b. All traffic enter user defined chain KUBE-FW-4E7KSV2ABIFJRAUZ will be firstly sent to user fined chain KUBE-MARK-MASQ, then KUBE-SVC-4E7KSV2ABIFJRAUZ, eventually if nothing matched, KUBE-MARK-DROP
  254. control-plane-mjnp4:/home/eccd # iptables -t nat -S|grep KUBE-FW-4E7KSV2ABIFJRAUZ
  255. -N KUBE-FW-4E7KSV2ABIFJRAUZ
  256. -A KUBE-FW-4E7KSV2ABIFJRAUZ -m comment --comment "ingress-nginx/ingress-nginx:https loadbalancer IP" -j KUBE-MARK-MASQ <-next step i
  257. -A KUBE-FW-4E7KSV2ABIFJRAUZ -m comment --comment "ingress-nginx/ingress-nginx:https loadbalancer IP" -j KUBE-SVC-4E7KSV2ABIFJRAUZ <-next step ii
  258. -A KUBE-FW-4E7KSV2ABIFJRAUZ -m comment --comment "ingress-nginx/ingress-nginx:https loadbalancer IP" -j KUBE-MARK-DROP <-next step iii
  259. -A KUBE-SERVICES -d 10.33.151.160/32 -p tcp -m comment --comment "ingress-nginx/ingress-nginx:https loadbalancer IP" -m tcp --dport 443 -j KUBE-FW-4E7KSV2ABIFJRAUZ
  260. control-plane-mjnp4:/home/eccd # 
  262. i. all traffic sent to chain KUBE-MARK-MASQ will be set with mark 0x4000/0x4000
  263. -A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
  265. Chain KUBE-MARK-MASQ (252 references)
  266. target     prot opt source               destination         
  267. MARK       all  --  anywhere             anywhere             MARK or 0x4000
  269. ii. All traffic with processed in chain KUBE-MARK-MASQ will be processed by the next rule, jump to chain KUBE-SVC-4E7KSV2ABIFJRAUZ . 
  270. control-plane-mjnp4:/home/eccd # iptables -t nat -S|grep KUBE-SVC-4E7KSV2ABIFJRAUZ | grep -v "j KUBE-SVC-4E7KSV2ABIFJRAUZ"
  271. -N KUBE-SVC-4E7KSV2ABIFJRAUZ
  272. -A KUBE-SVC-4E7KSV2ABIFJRAUZ -m comment --comment "ingress-nginx/ingress-nginx:https" -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-VNARWU5QWGSG2GUG <25% traffic will be load balanced to chain KUBE-SEP-VNARWU5QWGSG2GUG - remaining three nginx ingress controller pods>
  273. -A KUBE-SVC-4E7KSV2ABIFJRAUZ -m comment --comment "ingress-nginx/ingress-nginx:https" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-JRQUMAVVXJLRS3O3 <33.3% remaining traffic will be load balanced to chain KUBE-SEP-JRQUMAVVXJLRS3O3 - remaining two nginx ingress controller pods>
  274. -A KUBE-SVC-4E7KSV2ABIFJRAUZ -m comment --comment "ingress-nginx/ingress-nginx:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ILQWMWXVY6HNWZZT <50% remaining traffic will be load balanced to chain KUBE-SEP-ILQWMWXVY6HNWZZT- remaining one nginx ingress controller pods>
  275. -A KUBE-SVC-4E7KSV2ABIFJRAUZ -m comment --comment "ingress-nginx/ingress-nginx:https" -j KUBE-SEP-IM3OJF4BZQHSU4FK
  276. <remaining stream will be certainly sent to chain KUBE-SEP-IM3OJF4BZQHSU4FK - traffic are evenly load balanced to all four nginx ingress controller pods>
  277. control-plane-mjnp4:/home/eccd # 
  279. 1) Traffic entered KUBE-SEP-VNARWU5QWGSG2GUG, it's DNAT'ed to 192.168.103.142:443 since it's the above other three chains(nginx pods)
  280. control-plane-mjnp4:/home/eccd # iptables -t nat -S|grep KUBE-SEP-VNARWU5QWGSG2GUG | grep -v "j KUBE-SEP-VNARWU5QWGSG2GUG"
  281. -N KUBE-SEP-VNARWU5QWGSG2GUG
  282. -A KUBE-SEP-VNARWU5QWGSG2GUG -s 192.168.103.142/32 -m comment --comment "ingress-nginx/ingress-nginx:https" -j KUBE-MARK-MASQ 
  283. -A KUBE-SEP-VNARWU5QWGSG2GUG -p tcp -m comment --comment "ingress-nginx/ingress-nginx:https" -m tcp -j DNAT --to-destination 192.168.103.142:443 
  284. control-plane-mjnp4:/home/eccd # 
  285. control-plane-mjnp4:/home/eccd # iptables -t nat -S|grep KUBE-SEP-JRQUMAVVXJLRS3O3 | grep -v "j KUBE-SEP-JRQUMAVVXJLRS3O3"
  286. -N KUBE-SEP-JRQUMAVVXJLRS3O3
  287. -A KUBE-SEP-JRQUMAVVXJLRS3O3 -s 192.168.107.128/32 -m comment --comment "ingress-nginx/ingress-nginx:https" -j KUBE-MARK-MASQ
  288. -A KUBE-SEP-JRQUMAVVXJLRS3O3 -p tcp -m comment --comment "ingress-nginx/ingress-nginx:https" -m tcp -j DNAT --to-destination 192.168.107.128:443
  289. control-plane-mjnp4:/home/eccd # 
  290. control-plane-mjnp4:/home/eccd # iptables -t nat -S|grep KUBE-SEP-ILQWMWXVY6HNWZZT | grep -v "j KUBE-SEP-ILQWMWXVY6HNWZZT"
  291. -N KUBE-SEP-ILQWMWXVY6HNWZZT
  292. -A KUBE-SEP-ILQWMWXVY6HNWZZT -s 192.168.176.202/32 -m comment --comment "ingress-nginx/ingress-nginx:https" -j KUBE-MARK-MASQ
  293. -A KUBE-SEP-ILQWMWXVY6HNWZZT -p tcp -m comment --comment "ingress-nginx/ingress-nginx:https" -m tcp -j DNAT --to-destination 192.168.176.202:443
  294. control-plane-mjnp4:/home/eccd # 
  295. control-plane-mjnp4:/home/eccd # iptables -t nat -S|grep KUBE-SEP-IM3OJF4BZQHSU4FK | grep -v "j KUBE-SEP-IM3OJF4BZQHSU4FK"
  296. -N KUBE-SEP-IM3OJF4BZQHSU4FK
  297. -A KUBE-SEP-IM3OJF4BZQHSU4FK -s 192.168.95.125/32 -m comment --comment "ingress-nginx/ingress-nginx:https" -j KUBE-MARK-MASQ
  298. -A KUBE-SEP-IM3OJF4BZQHSU4FK -p tcp -m comment --comment "ingress-nginx/ingress-nginx:https" -m tcp -j DNAT --to-destination 192.168.95.125:443
  299. control-plane-mjnp4:/home/eccd #
  301. Question, who owns those IP?
  303. 2) Traffic will be sent to one of nginx ingress controller pods
  305. <--------Second load balancing by NGINX - Layer 7 -------->
  306. nginx-ingress-controller will load balance the HTTP traffic e.g. based on the URL.
  308. eccd@control-plane-mjnp4:~> kubectl get po -n ingress-nginx -l app=ingress-nginx -o=custom-columns=NODE:.spec.nodeName,Name:.metadata.name,hostIP:.status.hostIP,podIP:.status.podIP
  309. NODE                     Name                                        hostIP        podIP
  310. pool1-868fc5689d-9m7m5   nginx-ingress-controller-5dff4dcd48-dfbnr   10.0.10.102   192.168.107.128
  311. control-plane-gwnnh      nginx-ingress-controller-5dff4dcd48-m69kc   10.0.10.109   192.168.95.125
  312. control-plane-cr2zh      nginx-ingress-controller-5dff4dcd48-rcgv8   10.0.10.110   192.168.176.202
  313. pool2-6859c57999-2t8hx   nginx-ingress-controller-5dff4dcd48-t8v2g   10.0.10.108   192.168.103.142
  314. eccd@control-plane-mjnp4:~> 
  316. 3) Enter NGINX ingress controller pod and check HTTP load balancing rules - traffic will be sent to service monitoring/eric-pm-server:9090:
  317. eccd@control-plane-mjnp4:~> kubectl exec nginx-ingress-controller-5dff4dcd48-dfbnr -n ingress-nginx -ti -- bash
  318. bash-5.0$ cat nginx.conf
  320. # Configuration checksum: 5612144526815823684
  322. upstream upstream_balancer {
  323. ### Attention!!!
  324. #
  325. # We no longer create "upstream" section for every backend.
  326. # Backends are handled dynamically using Lua. If you would like to debug
  327. # and see what backends ingress-nginx has in its memory you can
  328. # install our kubectl plugin https://kubernetes.github.io/ingress-nginx/kubectl-plugin.
  329. # Once you have the plugin you can use "kubectl ingress-nginx backends" command to
  330. # inspect current backends.
  331. #
  332. ###
  333. ## start server dc173prometheus1.cloud.k2.ericsson.se
  334. server {
  335. server_name dc173prometheus1.cloud.k2.ericsson.se ;
  337. listen 80  ;
  338. listen 442 proxy_protocol  ssl http2 ;
  340. set $proxy_upstream_name "-";
  342. ssl_certificate_by_lua_block {
  343. certificate.call()
  344. }
  346. location / {
  348. set $namespace      "monitoring";
  349. set $ingress_name   "prometheus-server";
  350. set $service_name   "eric-pm-server";
  351. set $service_port   "9090";
  352. set $location_path  "/";
  354. rewrite_by_lua_block {
  355. lua_ingress.rewrite({
  356. force_ssl_redirect = false,
  357. ssl_redirect = true,
  358. force_no_ssl_redirect = false,
  359. use_port_in_redirects = false,
  360. })
  361. balancer.rewrite()
  362. plugins.run()
  363. }
  365. # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
  366. # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
  367. # other authentication method such as basic auth or external auth useless - all requests will be allowed.
  368. #access_by_lua_block {
  369. #}
  371. header_filter_by_lua_block {
  372. lua_ingress.header()
  373. plugins.run()
  374. }
  376. body_filter_by_lua_block {
  377. }
  379. log_by_lua_block {
  380. balancer.log()
  382. monitor.call()
  384. plugins.run()
  385. }
  387. port_in_redirect off;
  389. set $balancer_ewma_score -1;
  390. set $proxy_upstream_name "monitoring-eric-pm-server-9090";
  391. set $proxy_host          $proxy_upstream_name;
  392. set $pass_access_scheme  $scheme;
  394. set $pass_server_port    $server_port;
  396. set $best_http_host      $http_host;
  397. set $pass_port           $pass_server_port;
  399. set $proxy_alternative_upstream_name "";
  401. client_max_body_size                    1m;
  403. proxy_set_header Host                   $best_http_host;
  405. # Pass the extracted client certificate to the backend
  407. # Allow websocket connections
  408. proxy_set_header                        Upgrade           $http_upgrade;
  410. proxy_set_header                        Connection        $connection_upgrade;
  412. proxy_set_header X-Request-ID           $req_id;
  413. proxy_set_header X-Real-IP              $remote_addr;
  415. proxy_set_header X-Forwarded-For        $remote_addr;
  417. proxy_set_header X-Forwarded-Host       $best_http_host;
  418. proxy_set_header X-Forwarded-Port       $pass_port;
  419. proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
  421. proxy_set_header X-Scheme               $pass_access_scheme;
  423. # Pass the original X-Forwarded-For
  424. proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
  426. # mitigate HTTPoxy Vulnerability
  427. # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
  428. proxy_set_header Proxy                  "";
  430. # Custom headers to proxied server
  432. proxy_connect_timeout                   5s;
  433. proxy_send_timeout                      60s;
  434. proxy_read_timeout                      60s;
  436. proxy_buffering                         off;
  437. proxy_buffer_size                       4k;
  438. proxy_buffers                           4 4k;
  440. proxy_max_temp_file_size                1024m;
  442. proxy_request_buffering                 on;
  443. proxy_http_version                      1.1;
  445. proxy_cookie_domain                     off;
  446. proxy_cookie_path                       off;
  448. # In case of errors try the next upstream server before returning an error
  449. proxy_next_upstream                     error timeout;
  450. proxy_next_upstream_timeout             0;
  451. proxy_next_upstream_tries               3;
  453. proxy_pass http://upstream_balancer;
  455. proxy_redirect                          off;
  457. }
  459. }
  460. ## end server dc173prometheus1.cloud.k2.ericsson.se
  461. 4) Check service eric-pm-server definition
  462. eccd@control-plane-wfb88:~> kubectl get ing prometheus-server  -n monitoring -o yaml
  463. apiVersion: extensions/v1beta1
  464. kind: Ingress
  465. metadata:
  466. ...
  467. spec:
  468. rules:
  469. - host: dc173prometheus1.cloud.k2.ericsson.se
  470. http:
  471. paths:
  472. - backend:
  473. serviceName: eric-pm-server
  474. servicePort: 9090
  475. path: /
  476. pathType: ImplementationSpecific
  477. status:
  478. loadBalancer:
  479. ingress:
  480. - ip: 10.0.10.102
  481. - ip: 10.0.10.103
  482. - ip: 10.0.10.104
  483. - ip: 10.0.10.108
  484. 5) Traffic then will be forwarded to service clusterIP 10.107.185.250.
  485. eccd@control-plane-mjnp4:~> kubectl get svc eric-pm-server -n monitoring -o yaml
  486. spec:
  487. clusterIP: 10.107.185.250
  488. ports:
  489. - name: http
  490. port: 9090
  491. protocol: TCP
  492. targetPort: 9090
  493. selector:
  494. app: eric-pm-server
  495. component: server
  496. release: pm-server
  497. sessionAffinity: None
  498. type: ClusterIP
  499. status:
  500. loadBalancer: {}
  501. control-plane-mjnp4:/home/eccd # iptables -t nat -S | grep 10.107.185.250
  502. -A KUBE-SERVICES ! -s 192.168.0.0/16 -d 10.107.185.250/32 -p tcp -m comment --comment "monitoring/eric-pm-server:http cluster IP" -m tcp --dport 9090 -j KUBE-MARK-MASQ
  503. -A KUBE-SERVICES -d 10.107.185.250/32 -p tcp -m comment --comment "monitoring/eric-pm-server:http cluster IP" -m tcp --dport 9090 -j KUBE-SVC-6K4R6JTKTVNBU5UP
  504. control-plane-mjnp4:/home/eccd # 
  505. 6) First rule in chain KUBE-SERVICES is to set mark in KUBE-MARK-MASQ
  506. Second rule in chain KUBE-SERVICES will forward traffic to chain KUBE-SVC-6K4R6JTKTVNBU5UP
  507. control-plane-mjnp4:/home/eccd # iptables -t nat -S | grep KUBE-SVC-6K4R6JTKTVNBU5UP | grep -v "j KUBE-SVC-6K4R6JTKTVNBU5UP"
  508. -N KUBE-SVC-6K4R6JTKTVNBU5UP
  509. -A KUBE-SVC-6K4R6JTKTVNBU5UP -m comment --comment "monitoring/eric-pm-server:http" -j KUBE-SEP-CNB2ICYJPHYJI3AY
  510. control-plane-mjnp4:/home/eccd #
  511. 7) traffic enters chain KUBE-SEP-CNB2ICYJPHYJI3AY, will be sent to 192.168.22.247 which is pod IP.
  512. control-plane-mjnp4:/home/eccd # iptables -t nat -S | grep KUBE-SEP-CNB2ICYJPHYJI3AY | grep -v "j KUBE-SEP-CNB2ICYJPHYJI3AY"
  513. -N KUBE-SEP-CNB2ICYJPHYJI3AY
  514. -A KUBE-SEP-CNB2ICYJPHYJI3AY -s 192.168.22.247/32 -m comment --comment "monitoring/eric-pm-server:http" -j KUBE-MARK-MASQ
  515. -A KUBE-SEP-CNB2ICYJPHYJI3AY -p tcp -m comment --comment "monitoring/eric-pm-server:http" -m tcp -j DNAT --to-destination 192.168.22.247:9090
  516. control-plane-mjnp4:/home/eccd # 
  518. eccd@control-plane-mjnp4:~> kubectl get po -l app=eric-pm-server -A -o wide
  519. NAMESPACE    NAME               READY   STATUS    RESTARTS   AGE     IP                NODE                     NOMINATED NODE   READINESS GATES
  520. evnfm        eric-pm-server-0   2/2     Running   0          4h29m   192.168.188.101   pool2-6859c57999-mzv2d   <none>           <none>
  521. monitoring   eric-pm-server-0   2/2     Running   0          2d1h    192.168.22.247    pool2-6859c57999-7hfs2   <none>           <none>
  522. eccd@control-plane-mjnp4:~> 
  524. Since the server in question is in monitoring namespace, the correct PM server shall be the second one.
  526. <--------Third load balancing by K8s service - round-robin using iptables mode-------->
  527. There is only one eric-pm-server pod so there is no round-robin load balancing.
  529. eccd@control-plane-mjnp4:~> kubectl get no -o wide | grep pool2-6859c57999-7hfs2 
  530. pool2-6859c57999-7hfs2   Ready    worker   19d   v1.18.8   10.0.10.105   <none>        SUSE Linux Enterprise Server 15 SP1   4.12.14-197.51-default   docker://19.3.11
  531. eccd@control-plane-mjnp4:~> ssh 10.0.10.105 -q
  532. Last login: Mon Sep 21 16:27:26 2020 from 10.0.10.101
  533. eccd@pool2-6859c57999-7hfs2:~> sudo su
  534. pool2-6859c57999-7hfs2:/home/eccd # iptables -t nat -S | grep 192.168.22.247
  535. -A KUBE-SEP-CNB2ICYJPHYJI3AY -s 192.168.22.247/32 -m comment --comment "monitoring/eric-pm-server:http" -j KUBE-MARK-MASQ
  536. -A KUBE-SEP-CNB2ICYJPHYJI3AY -p tcp -m comment --comment "monitoring/eric-pm-server:http" -m tcp -j DNAT --to-destination 192.168.22.247:9090
  537. pool2-6859c57999-7hfs2:/home/eccd #
  539. Traffic enters the correct pod and we will now look at the return path.
  541. iii. Traffic will be set to mask 0x8000 if not matched by any of above rules. 
  542. control-plane-mjnp4:/home/eccd # iptables -t nat -S|grep KUBE-MARK-DROP | grep -v "j KUBE-MARK-DROP"
  543. -N KUBE-MARK-DROP
  544. -A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
  545. control-plane-mjnp4:/home/eccd # 
  547. eccd@control-plane-wfb88:~> kubectl get ing -A
  548. NAMESPACE     NAME                                  CLASS    HOSTS                                               ADDRESS                                           PORTS     AGE
  549.         monitoring    prometheus-server                     <none>   dc173prometheus1.cloud.k2.ericsson.se               10.0.10.102,10.0.10.103,10.0.10.104,10.0.10.108   80        28d

==============

return path

Return path will be using default route if there is no explicit route or existing route for dst=10.210.150.87.
1. Start point - return traffic with source IP 192.168.22.247/32 will be sent to KUBE-MARK-MASQ chain
pool2-6859c57999-7hfs2:/home/eccd # iptables -t nat -S | grep 192.168.22.247
-A KUBE-SEP-CNB2ICYJPHYJI3AY -s 192.168.22.247/32 -m comment —comment “monitoring/eric-pm-server:http” -j KUBE-MARK-MASQ
-A KUBE-SEP-CNB2ICYJPHYJI3AY -p tcp -m comment —comment “monitoring/eric-pm-server:http” -m tcp -j DNAT —to-destination 192.168.22.247:9090
pool2-6859c57999-7hfs2:/home/eccd #
2. All return traffic are set with mask 0x4000.
A KUBE-MARK-MASQ -j MARK —set-xmark 0x4000/0x4000

Chain KUBE-MARK-MASQ (242 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             MARK or 0x4000
3. traffic with UBE-MARK-MASQ mark will return to chain CNB2ICYJPHYJI3AY and check what's next rule - no more rules in chain KUBE-SEP-CNB2ICYJPHYJI3AY
pool2-6859c57999-7hfs2:/home/eccd # iptables -t nat -S | grep KUBE-SEP-CNB2ICYJPHYJI3AY 
-N KUBE-SEP-CNB2ICYJPHYJI3AY
-A KUBE-SEP-CNB2ICYJPHYJI3AY -s 192.168.22.247/32 -m comment --comment "monitoring/eric-pm-server:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-CNB2ICYJPHYJI3AY -p tcp -m comment --comment "monitoring/eric-pm-server:http" -m tcp -j DNAT --to-destination 192.168.22.247:9090
-A KUBE-SVC-6K4R6JTKTVNBU5UP -m comment --comment "monitoring/eric-pm-server:http" -j KUBE-SEP-CNB2ICYJPHYJI3AY
pool2-6859c57999-7hfs2:/home/eccd #
4. KUBE-MARK-MASQ adds a Netfilter mark to packets originated from the eric-pm-serve service which destined outside the cluster’s network. Packets with this mark will be altered in a POSTROUTING rule to use source network address translation (SNAT) with the node’s IP address as their source IP address.

From <https://www.stackrox.com/post/2020/01/kubernetes-networking-demystified/> 

5. Destination IP is not a the explicit routing entries, use default route
pool2-6859c57999-7hfs2:/home/eccd # ip r | grep 10.210.150.87
pool2-6859c57999-7hfs2:/home/eccd # 
pool2-6859c57999-7hfs2:/home/eccd # ip r | grep default
default via 10.33.152.65 dev ecfe_traf1 proto static metric 804 
pool2-6859c57999-7hfs2:/home/eccd # 
pool2-6859c57999-7hfs2:/home/eccd # ip a show ecfe_traf1
12: ecfe_traf1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 36:1f:d1:91:3a:16 brd ff:ff:ff:ff:ff:ff
    inet 10.33.152.74/27 brd 10.33.152.95 scope global noprefixroute ecfe_traf1
       valid_lft forever preferred_lft forever
pool2-6859c57999-7hfs2:/home/eccd # 


At this stage, IP-SA= 10.33.152.74, IP-DA=10.210.150.87. 
Since the packet ingress at DC173_CCD1_om_vr, the egress traffic must go back to the same VRF.
CNIS 1.0 default route belongs to application, therefore the egress traffic shall be forwarded via ccd_ecfe_om interface.

Ingress and egress traffic are routed via different VRF, TCP session won't be established.
Add route(temporary workaround, not persistent):
for i in `kubectl get no -o json | jq -r '.items[].status.addresses[] | select(.type=="InternalIP") | .address'` ; do ssh $i -q sudo ip route add <web-browser-ip>/32 via <ccd_ecfe-SLX-om_vr-ve_anycast> ; done

Egress traffic will be
Prometheus clusterip(iptables+conntrack) -> nginx ingress controller pod(-> srcIP=ingress-pod-ip) -> server host iptables hosting nginx controller pod(-> srcIP=ingress-ip) > lookup its host OS routing table -> ecfe gw(anycast GW IP in SLX) -> 10.210.150.87
src-IP = Promethers pod