Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 6957eae

Browse files
pankaj101Apankaj101A
authored
chore: docs update & disable fluentbit github checks (#828)
* chore: docs update * chore: docs update * chore: docs update * chore: docs update * chore: test according to otel * chore: test according to otel
1 parent d65f608 commit 6957eae

7 files changed

Lines changed: 345 additions & 60 deletions

File tree

‎.github/workflows/pull_requests.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ jobs:
5555
timeout-minutes: 15
5656
strategy:
5757
matrix:
58-
sidecar: [ fluentbit,otelcol ]
58+
sidecar: [ otelcol ]
5959
steps:
6060
- uses: actions/checkout@v4
6161
- name: Setup go
@@ -83,7 +83,7 @@ jobs:
8383
timeout-minutes: 15
8484
strategy:
8585
matrix:
86-
sidecar: [fluentbit,otelcol]
86+
sidecar: [otelcol]
8787
steps:
8888
- uses: actions/checkout@v4
8989
- name: Setup go
@@ -111,7 +111,7 @@ jobs:
111111
timeout-minutes: 15
112112
strategy:
113113
matrix:
114-
sidecar: [fluentbit,otelcol]
114+
sidecar: [otelcol]
115115
steps:
116116
- uses: actions/checkout@v4
117117
- name: Setup go
@@ -139,7 +139,7 @@ jobs:
139139
timeout-minutes: 15
140140
strategy:
141141
matrix:
142-
sidecar: [fluentbit,otelcol]
142+
sidecar: [otelcol]
143143
steps:
144144
- uses: actions/checkout@v4
145145
- name: Setup go

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,8 @@ or when it outputs additional logs to a file instead (eg. the gc.log).
3636
It [tails](https://en.wikipedia.org/wiki/Tail_(Unix)) the files inside Kubernetes Pods,
3737
handling situations like the file not being there when tailing starts, tailing multiple files, rotating files, etc.
3838
39-
It uses [Fluent Bit](https://fluentbit.io/) under the hood, benefiting from its performance.
39+
It uses [Sumologic Collector](https://www.sumologic.com/help/docs/send-data/opentelemetry-collector/) version 0.19.0 onwards.
40+
Before that [Fluent Bit](https://fluentbit.io/) was used.
4041
4142
For more information about cluster-level logging architecture please read Kubernetes
4243
[documentation](https://kubernetes.io/docs/concepts/cluster-administration/logging/#cluster-level-logging-architectures).

‎helm/tailing-sidecar-operator/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,8 @@ In order to override tailing sidecar configuration, the following properties may
5656
```yaml
5757
sidecar:
5858
config:
59-
mountPath: /fluent-bit/etc/
60-
content:
59+
mountPath: /etc/otel #for version>=0.19 .0 or /fluent-bit/etc/ for version <=0.18.1
60+
content: # applicable for version <=0.18.1
6161
file-1.conf: |
6262
content of file-1.conf
6363
file-2.conf: |
@@ -66,4 +66,4 @@ sidecar:
6666
6767
The above configuration is going to create `file-1.conf` and `file-2.conf` in `/fluent-bit/etc/` directory.
6868

69-
**All existing content of `/fluent-bit/etc/` directory will be replaced with the `sidecar.config.content`.**
69+
**All existing content of `/fluent-bit/etc/` directory will be replaced with the `sidecar.config.content`.**

‎helm/tests/values.withOtelcolCustomConfiguration.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ sidecar:
3535
extensions:
3636
text_encoding:
3737
file_storage:
38-
directory: /var/lib/otc
38+
directory: ${OTEL_FILE_STORAGE_PATH}
3939
4040
service:
4141
extensions:
@@ -46,7 +46,7 @@ sidecar:
4646
level: none
4747
logs:
4848
output_paths:
49-
- /var/log/otelcol.log
49+
- ${SIDECAR_OTEL_LOG_PATH}/otelcol.log
5050
pipelines:
5151
logs:
5252
exporters: [file]

‎sidecar/otelcol/README.md‎

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
# Tailing sidecar container image
2+
3+
**tailing sidecar container image** is a an image which can be used to manually extend Pods by [streaming sidecar containers](https://kubernetes.io/docs/concepts/cluster-administration/logging/#streaming-sidecar-container).
4+
5+
## Getting Started
6+
7+
To understand benefits of using tailing sidecar see example below.
8+
9+
### Extend Pod by adding tailing sidecars
10+
11+
Assuming that container writes logs to two different files and Pod has this specification:
12+
13+
```yaml
14+
apiVersion: v1
15+
kind: Pod
16+
metadata:
17+
name: example-with-tailling-sidecars
18+
spec:
19+
containers:
20+
- name: count
21+
image: busybox
22+
args:
23+
- /bin/sh
24+
- -c
25+
- >
26+
i=0;
27+
while true;
28+
do
29+
echo "example1: $i $(date)" >> /var/log/example1.log;
30+
echo "example2: $i $(date)" >> /var/log/example2.log;
31+
i=$((i+1));
32+
sleep 1;
33+
done
34+
volumeMounts:
35+
- name: varlog
36+
mountPath: /var/log
37+
volumes:
38+
- name: varlog
39+
emptyDir: {}
40+
```
41+
42+
Pod can be extended by adding tailing sidecar containers for easier log access:
43+
44+
```yaml
45+
apiVersion: v1
46+
kind: Pod
47+
metadata:
48+
name: example-with-tailling-sidecars
49+
spec:
50+
containers:
51+
- name: count
52+
image: busybox
53+
args:
54+
- /bin/sh
55+
- -c
56+
- >
57+
i=0;
58+
while true;
59+
do
60+
echo "example1: $i $(date)" >> /var/log/example1.log;
61+
echo "example2: $i $(date)" >> /var/log/example2.log;
62+
i=$((i+1));
63+
sleep 1;
64+
done
65+
volumeMounts:
66+
- name: varlog
67+
mountPath: /var/log
68+
- name: sidecar1
69+
image: ghcr.io/sumologic/tailing-sidecar:latest
70+
env:
71+
- name: PATH_TO_TAIL
72+
value: /var/log/example1.log
73+
- name: LOG_LEVEL
74+
value: warning
75+
volumeMounts:
76+
- name: varlog
77+
mountPath: /var/log
78+
- name: volume-sidecar-1
79+
mountPath: /tailing-sidecar/var
80+
- name: sidecar2
81+
image: ghcr.io/sumologic/tailing-sidecar:latest
82+
env:
83+
- name: PATH_TO_TAIL
84+
value: /var/log/example2.log
85+
- name: LOG_LEVEL
86+
value: warning
87+
volumeMounts:
88+
- name: varlog
89+
mountPath: /var/log
90+
- name: volume-sidecar-2
91+
mountPath: /tailing-sidecar/var
92+
volumes:
93+
- name: varlog
94+
emptyDir: {}
95+
- name: volume-sidecar-1
96+
hostPath:
97+
path: /var/log/sidecar1
98+
type: DirectoryOrCreate
99+
- name: volume-sidecar-2
100+
hostPath:
101+
path: /var/log/sidecar2
102+
type: DirectoryOrCreate
103+
```
104+
105+
Notice that tailing sidecar containers are configured through two environmental variables:
106+
107+
- `PATH_TO_TAIL` - pattern specifying a log file or multiple ones through the use of common wildcards,
108+
multiple patterns separated by commas are also allowed
109+
- `LOG_LEVEL` - verbosity level, by default 'warning' is set,
110+
allowed values: error, warning, info, debug, trace
111+
- `OTEL_FILE_STORAGE_PATH` - path to directory where filelog reciever stores data,
112+
- `SIDECAR_OTEL_LOG_PATH` - dir path for otel collector own logs. Logs will be in otel.log file inside this directory
113+
114+
115+
Try it!
116+
117+
```bash
118+
kubectl apply -f sidecar/examples/pod_with_tailing_sidecars.yaml
119+
```
120+
121+
And check logs:
122+
123+
```bash
124+
$ kubectl logs example-with-tailling-sidecars sidecar1
125+
example1: 0 Wed Jan 27 11:59:28 UTC 2021
126+
example1: 1 Wed Jan 27 11:59:29 UTC 2021
127+
example1: 2 Wed Jan 27 11:59:30 UTC 2021
128+
example1: 3 Wed Jan 27 11:59:31 UTC 2021
129+
...
130+
```
131+
132+
```bash
133+
$ kubectl logs example-with-tailling-sidecars sidecar2
134+
example2: 0 Wed Jan 27 11:59:28 UTC 2021
135+
example2: 1 Wed Jan 27 11:59:29 UTC 2021
136+
example2: 2 Wed Jan 27 11:59:30 UTC 2021
137+
example2: 3 Wed Jan 27 11:59:31 UTC 2021
138+
...
139+
```
140+
141+
## Run tailing sidecar in Docker container
142+
143+
To run tailing sidecar in Docker container define following variables:
144+
145+
- `DIR_TO_TAIL` - path to directory with files to read
146+
- `OTEL_FILE_STORAGE_PATH` - path to directory where filelog reciever stores data,
147+
- `FILES_PATTERN` - pattern to match files in directory specified as `DIR_TO_TAIL`
148+
- `TAILING_SIDECAR_IMAGE` - tailing sidecar Docker image
149+
- `SIDECAR_OTEL_LOG_PATH` - dir path for otel own logs. Logs will be in otel.log file inside this directory
150+
- `LOG_LEVEL` - verbosity level, by default 'warning' is set,
151+
allowed values: error, warning, info, debug, trace
152+
153+
e.g.
154+
155+
```bash
156+
export DIR_TO_TAIL="$PWD/examples"
157+
export OTEL_FILE_STORAGE_PATH="$PWD/var"
158+
export FILES_PATTERN="*.log"
159+
export TAILING_SIDECAR_IMAGE="ghcr.io/sumologic/tailing-sidecar:latest"
160+
export LOG_LEVEL="warning"
161+
export SIDECAR_OTEL_LOG_PATH="$PWD/var"
162+
```
163+
164+
And run tailing sidecar in Docker container:
165+
166+
```bash
167+
docker run --rm -it \
168+
-v ${DIR_TO_TAIL}:/tmp/host \
169+
-v ${$SIDECAR_OTEL_LOG_PATH}:/tailing-sidecar/var/log \
170+
-v ${OTEL_FILE_STORAGE_PATH}:/tailing-sidecar/var \
171+
--env "PATH_TO_TAIL=/tmp/host/${FILES_PATTERN}" \
172+
--env "SIDECAR_OTEL_LOG_PATH=$SIDECAR_OTEL_LOG_PATH" \
173+
--env "OTEL_FILE_STORAGE_PATH=$OTEL_FILE_STORAGE_PATH" \
174+
--env "LOG_LEVEL=${LOG_LEVEL}" ${TAILING_SIDECAR_IMAGE}
175+
```
176+
177+
## Build and run tailing sidecar in Docker container
178+
179+
To build and run Docker container with tailing sidecar to tail files in `$PWD/examples`
180+
which match pattern `*.log` and save Fluent Bit database in `$PWD/var`:
181+
182+
```bash
183+
make run \
184+
TAG=sidecar:dev \
185+
DIR_TO_TAIL="$PWD/examples" \
186+
FILES_PATTERN="*.log" \
187+
OTEL_FILE_STORAGE_PATH="$PWD/var" \
188+
SIDECAR_OTEL_LOG_PATH="$PWD/var" \
189+
LOG_LEVEL="warning"
190+
```
191+
192+
## Build and push tailing sidecar to Docker registry
193+
194+
To build Docker image with tailing sidecar:
195+
196+
```bash
197+
make build TAG=<DOCKER_IMAGE_TAG>
198+
```
199+
200+
e.g.
201+
202+
```bash
203+
make build TAG=localhost:32000/sumologic/tailing-sidecar:latest
204+
```
205+
206+
To push Docker image to container registry:
207+
208+
```bash
209+
make push TAG=<DOCKER_IMAGE_TAG>
210+
```
211+
212+
e.g.
213+
214+
```bash
215+
make push TAG=localhost:32000/sumologic/tailing-sidecar:latest
216+
```
217+
218+
## Testing in Vagrant environment
219+
220+
### Prerequisites
221+
222+
Please install the following:
223+
224+
- [VirtualBox](https://www.virtualbox.org/)
225+
- [Vagrant](https://www.vagrantup.com/)
226+
- [vagrant-disksize](https://github.com/sprotheroe/vagrant-disksize) plugin
227+
228+
### Setting up
229+
230+
Start and provision the Vagrant environment:
231+
232+
```bash
233+
vagrant up
234+
```
235+
236+
Connect to virtual machine:
237+
238+
```bash
239+
vagrant ssh
240+
```
241+
242+
### Build and run tailing sidecar
243+
244+
Build and push docker image to local container registry:
245+
246+
```bash
247+
/tailing-sidecar/sidecar/otelcol/Makefile
248+
```
249+
250+
Run example Pod:
251+
252+
```bash
253+
kubectl apply -f /tailing-sidecar/sidecar/otel/examples/pod_with_tailing_sidecars.yaml
254+
```

‎sidecar/otelcol/examples/pod_with_tailing_sidecars.yaml‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
apiVersion: v1
22
kind: Pod
33
metadata:
4-
name: example-with-otelcol-tailing-sidecars
4+
name: example-with-tailling-sidecars
55
spec:
66
containers:
77
- name: count
@@ -22,30 +22,44 @@ spec:
2222
- name: varlog
2323
mountPath: /var/log
2424
- name: sidecar1
25-
image: otel-side-car:latest
26-
imagePullPolicy: Never
25+
image: ghcr.io/sumologic/tailing-sidecar:latest
2726
env:
2827
- name: PATH_TO_TAIL
2928
value: /var/log/example1.log
29+
- name: OTEL_FILE_STORAGE_PATH
30+
value: /var/sidecar1/
31+
- name: SIDECAR_OTEL_LOG_PATH
32+
value: /var/sidecar1/
3033
volumeMounts:
3134
- name: varlog
3235
mountPath: /var/log
3336
- name: volume-sidecar-1
3437
mountPath: /tailing-sidecar/var
38+
- name: otel-volume-1
39+
mountPath: /var/sidecar1/
3540
- name: sidecar2
36-
image: otel-side-car:latest
37-
imagePullPolicy: Never
41+
image: ghcr.io/sumologic/tailing-sidecar:latest
3842
env:
3943
- name: PATH_TO_TAIL
4044
value: /var/log/example2.log
45+
- name: OTEL_FILE_STORAGE_PATH
46+
value: /var/sidecar2/
47+
- name: SIDECAR_OTEL_LOG_PATH
48+
value: /var/sidecar2/
4149
volumeMounts:
4250
- name: varlog
4351
mountPath: /var/log
52+
- name: otel-volume-2
53+
mountPath: /var/sidecar2/
4454
- name: volume-sidecar-2
4555
mountPath: /tailing-sidecar/var
4656
volumes:
4757
- name: varlog
4858
emptyDir: {}
59+
- name: otel-volume-1
60+
emptyDir: { }
61+
- name: otel-volume-2
62+
emptyDir: { }
4963
- name: volume-sidecar-1
5064
hostPath:
5165
path: /var/log/sidecar1

0 commit comments

Comments
 (0)