Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Sep 13, 2022. It is now read-only.

Commit f9f5190

Browse files
quobyte-alekquobyte-alek
committed
Client DaemonSet redefinitions
Use namespaces to increase the client pod isolation.
1 parent b099ddd commit f9f5190

File tree

4 files changed

+181
-116
lines changed

4 files changed

+181
-116
lines changed

‎client_quick_setup.md‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,11 @@ EOF
1919
$ systemctl daemon-reload && systemctl restart docker
2020
```
2121

22+
Also make sure your Kubernetes cluster runs with the `MountPropagation` feature
23+
gate enabled for kubelets as well as kube-apiservers. Refer to a
24+
[feature support matrix](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/)
25+
for details.
26+
2227
## Client setup
2328
This guide assumes that you have a dedicated Quobyte server running and you
2429
want to provide access to Quobyte volumes to pods running in Kubernetes.
@@ -27,6 +32,17 @@ To access a Quobyte volume a pod has to run on a Kubernetes node which has a
2732
Quobyte client running. The client runs inside of a Pod and makes the Quobyte
2833
storage accessible to other pods.
2934

35+
We provide two different DaemonSet definitions: the [recommended one](deploy/client-ds.yaml)
36+
and [legacy one](deploy/client-ds-legacy.yaml). The former can be used only with
37+
explicit user and group mapping. The latter allows you to resolve user and group
38+
identity using a host Name Service Switch (NSS), which we do not recommend as
39+
this approach poses certain security and system stability risks. We provide the
40+
legacy DaemonSet configuration for backward compatibility reasons only. If using
41+
identity mapping feature is not suitable for your deployment scenario we
42+
recommend you to create a customized version of our client Docker image. This
43+
way you can apply necessary changes in the container's NSS facility to meet your
44+
requirements.
45+
3046
Quobyte clients run in the quobyte namespace.
3147
```bash
3248
$ cd deploy
@@ -37,11 +53,9 @@ To connect to Quobyte, the client needs to resolve the address of the registry.
3753
It is configured in the client-ds.yaml DaemonSet definition:
3854
```yaml
3955
env:
40-
- name: QUOBYTE_REGISTRY
41-
value: registry.quobyte
42-
```
43-
If you have a QNS (Quobyte Name Service), the value for QUOBYTE_REGISTRY might
44-
look like `<qnsid>.myquobyte.net`
56+
- name: QUOBYTE_REGISTRY value: registry.quobyte ``` If you have a QNS
57+
(Quobyte Name Service), the value for QUOBYTE_REGISTRY might look like
58+
`<qnsid>.myquobyte.net`
4559

4660
If you have a certificate for the client, it is stored as a Secret and
4761
mounted into the client Pod as client.cfg.
@@ -58,8 +72,9 @@ or
5872
$ kubectl -n quobyte create -f client-certificate-ds.yaml
5973
```
6074

61-
The deployed DaemonSet starts client pods on all nodes marked as `quobyte_client`.
62-
This can either be done manually, or by using the Quobyte operator.
75+
The deployed DaemonSet starts client pods on all nodes marked as
76+
`quobyte_client`. This can either be done manually, or by using the Quobyte
77+
operator.
6378

6479
```bash
6580
$ kubectl label nodes <node-1> <node-n> quobyte_client="true"

‎deploy/client-ds.yaml‎

Lines changed: 14 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ spec:
2828
value: registry.quobyte
2929
- name: QUOBYTE_MOUNT_POINT
3030
# Note that the mount point has to be a subdir of the volume(Mount)
31-
value: /var/lib/kubelet/plugins/kubernetes.io~quobyte
31+
value: /mnt/kubernetes.io~quobyte
3232
- name: NODENAME
3333
valueFrom:
3434
fieldRef:
@@ -53,79 +53,34 @@ spec:
5353
- /bin/bash
5454
- -xec
5555
- |
56-
if [[ ! -f /etcfs/fuse.conf ]]; then
57-
echo "Copy fuse config to host"
58-
{ echo -e '# Copied from Quobyte Client Container\n'; cat /etc/fuse.conf; } > /etcfs/fuse.conf
59-
fi
60-
if [[ $(grep "^[^#]" /etcfs/fuse.conf | grep -c "user_allow_other") -eq 0 ]]; then
61-
echo "user_allow_other" >> /etcfs/fuse.conf
62-
fi
63-
if cut -d" " -f2 /etcfs/mtab | grep -q ${QUOBYTE_MOUNT_POINT}; then
64-
echo "found existing mount point on host. Trying to umount ${QUOBYTE_MOUNT_POINT}"
65-
if ! /bin/nsenter -t 1 -m -- /bin/umount -f ${QUOBYTE_MOUNT_POINT}; then
66-
/bin/nsenter -t 1 -m -- /bin/umount -l ${QUOBYTE_MOUNT_POINT}
67-
fi
68-
sleep 1 # give os the chance to clean up everything.
69-
else
70-
if ! [[ $(ls `dirname ${QUOBYTE_MOUNT_POINT}`|egrep "^`basename ${QUOBYTE_MOUNT_POINT}`$") ]]; then
71-
echo "mount point ${QUOBYTE_MOUNT_POINT} does not exist, creating it..."
72-
mkdir -p ${QUOBYTE_MOUNT_POINT}
73-
fi
56+
if cut -d" " -f2 /proc/self/mounts | grep -q ${QUOBYTE_MOUNT_POINT}; then
57+
umount -l ${QUOBYTE_MOUNT_POINT}
7458
fi
7559
60+
mkdir -p /root/.quobyte ${QUOBYTE_MOUNT_POINT}
61+
7662
# set the mount point immutable. As long as mount.quobyte does not run,
7763
# other processes cannot write data to this dir.
78-
chattr +i ${QUOBYTE_MOUNT_POINT}
79-
80-
# Until k8s mountPropagation is stable, nsenter is used for the client
81-
# to mount Quobyte into the host's mount namespace. By using nsenter, the client process
82-
# will also use the host's dns resolve mechanisms, which most probably will not include
83-
# the k8s internal name resolution. So the client is not able to resolve k8s internal 'registry.quobyte' anymore.
84-
# Hence, for k8s internal Quobyte clusters, we need to pre-resolve the current ips, and let the client
85-
# work with ips and not dns names.
86-
if [ "$QUOBYTE_REGISTRY" == "registry" ] || [ "$QUOBYTE_REGISTRY" == "registry.quobyte" ] ; then
87-
registry_ips=
88-
for ip in $(nslookup ${QUOBYTE_REGISTRY} | grep ${QUOBYTE_REGISTRY} -A2 | grep Address | cut -d':' -f2 | awk '{print $1}'); do
89-
registry_ips="$ip $registry_ips"
90-
done
91-
ADDR=$(echo $registry_ips | tr ' ' ',')
92-
echo "Assuming to connect to k8s hosted Quobyte cluster. Resolved DNS to: '$ADDR'"
93-
else
94-
ADDR=${QUOBYTE_REGISTRY}
95-
echo "Assuming to connect to an external Quobyte cluster. Client will resolve DNS for: '$ADDR'"
96-
fi
97-
98-
/bin/nsenter -t 1 --wd=. -m -- \
99-
lib/ld-linux-x86-64.so.2 \
100-
--library-path ./lib \
101-
/usr/bin/mount.quobyte --hostname ${NODENAME} --allow-usermapping-in-volumename \
102-
--http-port 55000 -f -l /dev/stdout -d ${QUOBYTE_CLIENT_LOG_LEVEL} ${OPTS} \
103-
${ADDR}/ ${QUOBYTE_MOUNT_POINT}
64+
chattr +i ${QUOBYTE_MOUNT_POINT} || \
65+
echo "WARNING: The local filesystem does not support IMMUTABLE flag"
10466
67+
/usr/bin/mount.quobyte --hostname ${NODENAME} \
68+
--allow-usermapping-in-volumename --http-port 55000 -f \
69+
-d ${QUOBYTE_CLIENT_LOG_LEVEL} -l /dev/stdout ${OPTS} \
70+
${QUOBYTE_REGISTRY}/ ${QUOBYTE_MOUNT_POINT}
10571
securityContext:
10672
privileged: true
10773
volumeMounts:
10874
- name: k8s-plugin-dir
109-
mountPath: /var/lib/kubelet/plugins/
110-
- name: etcfs
111-
mountPath: /etcfs
75+
mountPath: /mnt
76+
mountPropagation: Bidirectional
11277
lifecycle:
11378
preStop:
11479
exec:
115-
command:
116-
- /bin/bash
117-
- -xc
118-
- |
119-
if ! /bin/nsenter -t 1 -m -- /bin/umount -f ${QUOBYTE_MOUNT_POINT}; then
120-
/bin/nsenter -t 1 -m -- /bin/umount -l ${QUOBYTE_MOUNT_POINT}
121-
fi
122-
hostPID: true
80+
command: ["/bin/bash", "-xc", "umount -l ${QUOBYTE_MOUNT_POINT}"]
12381
nodeSelector:
12482
quobyte_client: "true"
12583
volumes:
12684
- name: k8s-plugin-dir
12785
hostPath:
12886
path: /var/lib/kubelet/plugins/
129-
- name: etcfs
130-
hostPath:
131-
path: /etc

‎deploy/legacy-client-ds.yaml‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
apiVersion: extensions/v1beta1
2+
kind: DaemonSet
3+
metadata:
4+
name: client
5+
namespace: quobyte
6+
spec:
7+
selector:
8+
matchLabels:
9+
role: client
10+
template:
11+
metadata:
12+
annotations:
13+
prometheus.io/scrape: 'true'
14+
prometheus.io/path: '/prometheus'
15+
prometheus.io/port: '55000'
16+
labels:
17+
role: client
18+
version: "2"
19+
spec:
20+
containers:
21+
- name: quobyte-client
22+
image: quay.io/quobyte/quobyte-client:2
23+
imagePullPolicy: Always
24+
env:
25+
- name: QUOBYTE_CLIENT_LOG_LEVEL
26+
value: INFO
27+
- name: QUOBYTE_REGISTRY
28+
value: registry.quobyte
29+
- name: QUOBYTE_MOUNT_POINT
30+
# Note that the mount point has to be a subdir of the volume(Mount)
31+
value: /var/lib/kubelet/plugins/kubernetes.io~quobyte
32+
- name: NODENAME
33+
valueFrom:
34+
fieldRef:
35+
fieldPath: spec.nodeName
36+
ports:
37+
- name: http-port
38+
containerPort: 55000
39+
hostPort: 55000
40+
protocol: TCP
41+
readinessProbe:
42+
timeoutSeconds: 5
43+
httpGet:
44+
port: 55000
45+
path: /
46+
livenessProbe:
47+
initialDelaySeconds: 30
48+
timeoutSeconds: 5
49+
httpGet:
50+
port: 55000
51+
path: /
52+
command:
53+
- /bin/bash
54+
- -xec
55+
- |
56+
if [[ ! -f /etcfs/fuse.conf ]]; then
57+
echo "Copy fuse config to host"
58+
{ echo -e '# Copied from Quobyte Client Container\n'; cat /etc/fuse.conf; } > /etcfs/fuse.conf
59+
fi
60+
if [[ $(grep "^[^#]" /etcfs/fuse.conf | grep -c "user_allow_other") -eq 0 ]]; then
61+
echo "user_allow_other" >> /etcfs/fuse.conf
62+
fi
63+
if cut -d" " -f2 /etcfs/mtab | grep -q ${QUOBYTE_MOUNT_POINT}; then
64+
echo "found existing mount point on host. Trying to umount ${QUOBYTE_MOUNT_POINT}"
65+
if ! /bin/nsenter -t 1 -m -- /bin/umount -f ${QUOBYTE_MOUNT_POINT}; then
66+
/bin/nsenter -t 1 -m -- /bin/umount -l ${QUOBYTE_MOUNT_POINT}
67+
fi
68+
sleep 1 # give os the chance to clean up everything.
69+
else
70+
if ! [[ $(ls `dirname ${QUOBYTE_MOUNT_POINT}`|egrep "^`basename ${QUOBYTE_MOUNT_POINT}`$") ]]; then
71+
echo "mount point ${QUOBYTE_MOUNT_POINT} does not exist, creating it..."
72+
mkdir -p ${QUOBYTE_MOUNT_POINT}
73+
fi
74+
fi
75+
76+
# set the mount point immutable. As long as mount.quobyte does not run,
77+
# other processes cannot write data to this dir.
78+
chattr +i ${QUOBYTE_MOUNT_POINT}
79+
80+
# Until k8s mountPropagation is stable, nsenter is used for the client
81+
# to mount Quobyte into the host's mount namespace. By using nsenter, the client process
82+
# will also use the host's dns resolve mechanisms, which most probably will not include
83+
# the k8s internal name resolution. So the client is not able to resolve k8s internal 'registry.quobyte' anymore.
84+
# Hence, for k8s internal Quobyte clusters, we need to pre-resolve the current ips, and let the client
85+
# work with ips and not dns names.
86+
if [ "$QUOBYTE_REGISTRY" == "registry" ] || [ "$QUOBYTE_REGISTRY" == "registry.quobyte" ] ; then
87+
registry_ips=
88+
for ip in $(nslookup ${QUOBYTE_REGISTRY} | grep ${QUOBYTE_REGISTRY} -A2 | grep Address | cut -d':' -f2 | awk '{print $1}'); do
89+
registry_ips="$ip $registry_ips"
90+
done
91+
ADDR=$(echo $registry_ips | tr ' ' ',')
92+
echo "Assuming to connect to k8s hosted Quobyte cluster. Resolved DNS to: '$ADDR'"
93+
else
94+
ADDR=${QUOBYTE_REGISTRY}
95+
echo "Assuming to connect to an external Quobyte cluster. Client will resolve DNS for: '$ADDR'"
96+
fi
97+
98+
/bin/nsenter -t 1 --wd=. -m -- \
99+
lib/ld-linux-x86-64.so.2 \
100+
--library-path ./lib \
101+
/usr/bin/mount.quobyte --hostname ${NODENAME} --allow-usermapping-in-volumename \
102+
--http-port 55000 -f -l /dev/stdout -d ${QUOBYTE_CLIENT_LOG_LEVEL} ${OPTS} \
103+
${ADDR}/ ${QUOBYTE_MOUNT_POINT}
104+
105+
securityContext:
106+
privileged: true
107+
volumeMounts:
108+
- name: k8s-plugin-dir
109+
mountPath: /var/lib/kubelet/plugins/
110+
- name: etcfs
111+
mountPath: /etcfs
112+
lifecycle:
113+
preStop:
114+
exec:
115+
command:
116+
- /bin/bash
117+
- -xc
118+
- |
119+
if ! /bin/nsenter -t 1 -m -- /bin/umount -f ${QUOBYTE_MOUNT_POINT}; then
120+
/bin/nsenter -t 1 -m -- /bin/umount -l ${QUOBYTE_MOUNT_POINT}
121+
fi
122+
hostPID: true
123+
nodeSelector:
124+
quobyte_client: "true"
125+
volumes:
126+
- name: k8s-plugin-dir
127+
hostPath:
128+
path: /var/lib/kubelet/plugins/
129+
- name: etcfs
130+
hostPath:
131+
path: /etc

‎operator/deploy/client.yaml‎

Lines changed: 14 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ spec:
4040
value: registry.quobyte
4141
- name: QUOBYTE_MOUNT_POINT
4242
# Note that the mount point has to be a subdir of the volume(Mount)
43-
value: /var/lib/kubelet/plugins/kubernetes.io~quobyte
43+
value: /mnt/kubernetes.io~quobyte
4444
- name: NODENAME
4545
valueFrom:
4646
fieldRef:
@@ -65,71 +65,35 @@ spec:
6565
- /bin/bash
6666
- -xec
6767
- |
68-
if [[ ! -f /etcfs/fuse.conf ]]; then
69-
echo "Copy fuse config to host"
70-
{ echo -e '# Copied from Quobyte Client Container\n'; cat /etc/fuse.conf; } > /etcfs/fuse.conf
71-
fi
72-
if [[ $(grep "^[^#]" /etcfs/fuse.conf | grep -c "user_allow_other") -eq 0 ]]; then
73-
echo "user_allow_other" >> /etcfs/fuse.conf
74-
fi
75-
if cut -d" " -f2 /etcfs/mtab | grep -q ${QUOBYTE_MOUNT_POINT}; then
76-
echo "found existing mount point on host. Trying to umount ${QUOBYTE_MOUNT_POINT}"
77-
/bin/nsenter -t 1 -m -- /bin/umount -f ${QUOBYTE_MOUNT_POINT}
78-
sleep 1 # give os the chance to clean up everything.
79-
else
80-
if ! [[ $(ls `dirname ${QUOBYTE_MOUNT_POINT}`|egrep "^`basename ${QUOBYTE_MOUNT_POINT}`$") ]]; then
81-
echo "mount point ${QUOBYTE_MOUNT_POINT} does not exist, creating it..."
82-
mkdir -p ${QUOBYTE_MOUNT_POINT}
83-
fi
68+
if cut -d" " -f2 /proc/self/mounts | grep -q ${QUOBYTE_MOUNT_POINT}; then
69+
umount -l ${QUOBYTE_MOUNT_POINT}
8470
fi
8571
72+
mkdir -p /root/.quobyte ${QUOBYTE_MOUNT_POINT}
73+
8674
# set the mount point immutable. As long as mount.quobyte does not run,
8775
# other processes cannot write data to this dir.
88-
chattr +i ${QUOBYTE_MOUNT_POINT}
76+
chattr +i ${QUOBYTE_MOUNT_POINT} || \
77+
echo "WARNING: The local filesystem does not support IMMUTABLE flag"
8978
90-
# Until k8s mountPropagation is stable, nsenter is used for the client
91-
# to mount Quobyte into the host's mount namespace. By using nsenter, the client process
92-
# will also use the host's dns resolve mechanisms, which most probably will not include
93-
# the k8s internal name resolution. So the client is not able to resolve k8s internal 'registry.quobyte' anymore.
94-
# Hence, for k8s internal Quobyte clusters, we need to pre-resolve the current ips, and let the client
95-
# work with ips and not dns names.
96-
if [ "$QUOBYTE_REGISTRY" == "registry" ] || [ "$QUOBYTE_REGISTRY" == "registry.quobyte" ] ; then
97-
registry_ips=
98-
for ip in $(nslookup ${QUOBYTE_REGISTRY} | grep ${QUOBYTE_REGISTRY} -A2 | grep Address | cut -d':' -f2 | awk '{print $1}'); do
99-
registry_ips="$ip $registry_ips"
100-
done
101-
ADDR=$(echo $registry_ips | tr ' ' ',')
102-
echo "Assuming to connect to k8s hosted Quobyte cluster. Resolved DNS to: '$ADDR'"
103-
else
104-
ADDR=${QUOBYTE_REGISTRY}
105-
echo "Assuming to connect to an external Quobyte cluster. Client will resolve DNS for: '$ADDR'"
106-
fi
107-
108-
/bin/nsenter -t 1 --wd=. -m -- \
109-
lib/ld-linux-x86-64.so.2 \
110-
--library-path ./lib \
111-
/usr/bin/mount.quobyte --hostname ${NODENAME} --allow-usermapping-in-volumename \
112-
--http-port 55000 -f -l /dev/stdout -d ${QUOBYTE_CLIENT_LOG_LEVEL} ${OPTS} \
113-
${ADDR}/ ${QUOBYTE_MOUNT_POINT}
79+
/usr/bin/mount.quobyte --hostname ${NODENAME} \
80+
--allow-usermapping-in-volumename --http-port 55000 -f \
81+
-d ${QUOBYTE_CLIENT_LOG_LEVEL} -l /dev/stdout ${OPTS} \
82+
${QUOBYTE_REGISTRY}/ ${QUOBYTE_MOUNT_POINT}
11483
securityContext:
11584
privileged: true
11685
volumeMounts:
11786
- name: k8s-plugin-dir
118-
mountPath: /var/lib/kubelet/plugins/
119-
- name: etcfs
120-
mountPath: /etcfs
87+
mountPath: /mnt
88+
mountPropagation: Bidirectional
12189
lifecycle:
12290
preStop:
12391
exec:
124-
command: ["/bin/bash", "-xc", "/bin/nsenter -t 1 -m -- /bin/umount -f ${QUOBYTE_MOUNT_POINT}"]
125-
hostPID: true
92+
command: ["/bin/bash", "-xc", "umount -l ${QUOBYTE_MOUNT_POINT}"]
12693
nodeSelector:
12794
# operator currently supports only single selector
12895
quobyte_client: "true"
12996
volumes:
13097
- name: k8s-plugin-dir
13198
hostPath:
13299
path: /var/lib/kubelet/plugins/
133-
- name: etcfs
134-
hostPath:
135-
path: /etc

0 commit comments

Comments
 (0)