docker容器C和其他主机B有网络连接,但是在容器宿主机A用netstat看不到。
现象:
B的服务日志可以看到来自A的请求
B上执行netstat看到有来自A的连接
A上执行netstat看不到来自B的连接
B是k8s节点
在上执行tcpdump dst A,可以看到发给A的网络包
说明网络连接是存在的,但netstat无法看到,是B上某个容器产生的
执行如下命令找出相关容器:
for cid in docker ps -q
;do pid=docker inspect -f "{{.State.Pid}}" $cid
;echo $cid,$pid;nsenter -t $pid -n netstat -anp|grep ${hosta-ip};done
docker inspect可以看到容器对应的进程号
如果容器没有相关命令,可以使用nsenter进入该进程执行netstat,从而查看到对应的网络连接和相关进程
nsenter -t $pid -n netstat
NSENTER(1) User Commands NSENTER(1)
NAME
nsenter - run program with namespaces of other processes
SYNOPSIS
nsenter [options] [program [arguments]]
DESCRIPTION
Enters the namespaces of one or more other processes and then executes the specified program. Enterable namespaces are:
mount namespace
Mounting and unmounting filesystems will not affect the rest of the system (CLONE_NEWNS flag), except for filesystems which are explicitly
marked as shared (with mount --make-shared; see /proc/self/mountinfo for the shared flag).
UTS namespace
Setting hostname or domainname will not affect the rest of the system. (CLONE_NEWUTS flag)
IPC namespace
The process will have an independent namespace for System V message queues, semaphore sets and shared memory segments. (CLONE_NEWIPC flag)
network namespace
The process will have independent IPv4 and IPv6 stacks, IP routing tables, firewall rules, the /proc/net and /sys/class/net directory trees,
sockets, etc. (CLONE_NEWNET flag)
PID namespace
Children will have a set of PID to process mappings separate from the nsenter process (CLONE_NEWPID flag). nsenter will fork by default if
changing the PID namespace, so that the new program and its children share the same PID namespace and are visible to each other. If --no-fork
is used, the new program will be exec'ed without forking.
user namespace
The process will have a distinct set of UIDs, GIDs and capabilities. (CLONE_NEWUSER flag)
See clone(2) for the exact semantics of the flags.
If program is not given, then ``${SHELL}'' is run (default: /bin/sh).
OPTIONS
-t, --target pid
Specify a target process to get contexts from. The paths to the contexts specified by pid are:
/proc/pid/ns/mnt the mount namespace
/proc/pid/ns/uts the UTS namespace
/proc/pid/ns/ipc the IPC namespace
/proc/pid/ns/net the network namespace
/proc/pid/ns/pid the PID namespace
/proc/pid/ns/user the user namespace
/proc/pid/root the root directory
/proc/pid/cwd the working directory respectively
-m, --mount[=file]
Enter the mount namespace. If no file is specified, enter the mount namespace of the target process. If file is specified, enter the mount
namespace specified by file.
-u, --uts[=file]
Enter the UTS namespace. If no file is specified, enter the UTS namespace of the target process. If file is specified, enter the UTS names‐
pace specified by file.
-i, --ipc[=file]
Enter the IPC namespace. If no file is specified, enter the IPC namespace of the target process. If file is specified, enter the IPC names‐
pace specified by file.
-n, --net[=file]
Enter the network namespace. If no file is specified, enter the network namespace of the target process. If file is specified, enter the
network namespace specified by file.
-p, --pid[=file]
Enter the PID namespace. If no file is specified, enter the PID namespace of the target process. If file is specified, enter the PID names‐
pace specified by file.
-U, --user[=file]
Enter the user namespace. If no file is specified, enter the user namespace of the target process. If file is specified, enter the user
namespace specified by file. See also the --setuid and --setgid options.
-G, --setgid gid
Set the group ID which will be used in the entered namespace and drop supplementary groups. nsenter(1) always sets GID for user namespaces,
the default is 0.
-S, --setuid uid
Set the user ID which will be used in the entered namespace. nsenter(1) always sets UID for user namespaces, the default is 0.
--preserve-credentials
Don't modify UID and GID when enter user namespace. The default is to drops supplementary groups and sets GID and UID to 0.
-r, --root[=directory]
Set the root directory. If no directory is specified, set the root directory to the root directory of the target process. If directory is
specified, set the root directory to the specified directory.
-w, --wd[=directory]
Set the working directory. If no directory is specified, set the working directory to the working directory of the target process. If direc‐
tory is specified, set the working directory to the specified directory.
-F, --no-fork
Do not fork before exec'ing the specified program. By default, when entering a PID namespace, nsenter calls fork before calling exec so that
any children will also be in the newly entered PID namespace.
-Z, --follow-context
Set the SELinux security context used for executing a new process according to already running process specified by --target PID. (The util-
linux has to be compiled with SELinux support otherwise the option is unavailable.)
-V, --version
Display version information and exit.
-h, --help
Display help text and exit.
SEE ALSO
setns(2), clone(2)