Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added a new directory for continuous pipeline configuration when make dist #1926

Merged
merged 3 commits into from
Dec 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 40 additions & 40 deletions docs/cn/installation/start-with-k8s.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,39 +8,39 @@

## 采集Kubernetes容器日志

1. 创建部署iLogtail的命名空间
1. 创建部署loongcollector的命名空间

将下面内容保存为ilogtail-ns.yaml
将下面内容保存为loongcollector-ns.yaml

```yaml {.line-numbers}
apiVersion: v1
kind: Namespace
metadata:
name: ilogtail
name: loongcollector
```

您也可以直接从下面的地址下载示例配置。

```bash
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-ns.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/loongcollector-ns.yaml
```

应用上述配置

```bash
kubectl apply -f ilogtail-ns.yaml
kubectl apply -f loongcollector-ns.yaml
```

2. 创建配置iLogtail的ConfigMap和Secret
2. 创建配置loongcollector的ConfigMap和Secret

将下面内容保存为ilogtail-user-configmap.yaml。该ConfigMap后续将作为配置目录挂载到iLogtail容器中,因此可包含多个采集配置。
将下面内容保存为loongcollector-user-configmap.yaml。该ConfigMap后续将作为配置目录挂载到loongcollector容器中,因此可包含多个采集配置。

```yaml {.line-numbers}
apiVersion: v1
kind: ConfigMap
metadata:
name: ilogtail-user-cm
namespace: ilogtail
name: loongcollector-user-cm
namespace: loongcollector
data:
nginx_stdout.yaml: |
enable: true
Expand Down Expand Up @@ -71,14 +71,14 @@
OnlyStdout: true
```

将下面内容保存为ilogtail-secret.yaml。该Secret为可选,当需要将日志写入SLS时会用到。
将下面内容保存为loongcollector-secret.yaml。该Secret为可选,当需要将日志写入SLS时会用到。

```yaml {.line-numbers}
apiVersion: v1
kind: Secret
metadata:
name: ilogtail-secret
namespace: ilogtail
name: loongcollector-secret
namespace: loongcollector
type: Opaque
data:
access_key_id: # base64 accesskey id if you want to flush to SLS
Expand All @@ -88,42 +88,42 @@
您也可以直接从下面的地址下载示例配置。

```bash
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-user-configmap.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-secret.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/loongcollector-user-configmap.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/loongcollector-secret.yaml
```

应用上述配置

```bash
kubectl apply -f ilogtail-user-configmap.yaml
kubectl apply -f ilogtail-secret.yaml
kubectl apply -f loongcollector-user-configmap.yaml
kubectl apply -f loongcollector-secret.yaml
```

3. 创建iLogtail DaemonSet
3. 创建loongcollector DaemonSet

将下面内容保存为ilogtail-daemonset.yaml。
将下面内容保存为loongcollector-daemonset.yaml。

```yaml {.line-numbers}
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ilogtail-ds
namespace: ilogtail
name: loongcollector-ds
namespace: loongcollector
labels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
spec:
selector:
matchLabels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
template:
metadata:
labels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
spec:
tolerations:
- operator: Exists # deploy on all nodes
containers:
- name: logtail
- name: loongcollector
env:
- name: ALIYUN_LOG_ENV_TAGS # add log tags from env
value: _node_name_|_node_ip_
Expand All @@ -137,24 +137,24 @@
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: cpu_usage_limit # iLogtail's self monitor cpu limit
- name: cpu_usage_limit # loongcollector's self monitor cpu limit
value: "1"
- name: mem_usage_limit # iLogtail's self monitor mem limit
- name: mem_usage_limit # loongcollector's self monitor mem limit
value: "512"
- name: default_access_key_id # accesskey id if you want to flush to SLS
valueFrom:
secretKeyRef:
name: ilogtail-secret
name: loongcollector-secret
key: access_key_id
optional: true
- name: default_access_key # accesskey secret if you want to flush to SLS
valueFrom:
secretKeyRef:
name: ilogtail-secret
name: loongcollector-secret
key: access_key
optional: true
image: >-
sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:latest
sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/loongcollector-community-edition/loongcollector:latest
imagePullPolicy: IfNotPresent
resources:
limits:
Expand All @@ -170,9 +170,9 @@
mountPropagation: HostToContainer
name: root
readOnly: true
- mountPath: /usr/local/ilogtail/checkpoint # for checkpoint between container restart
- mountPath: /usr/local/loongcollector/data # for checkpoint between container restart
name: checkpoint
- mountPath: /usr/local/ilogtail/config/local # mount config dir
- mountPath: /usr/local/loongcollector/conf/continuous_pipeline_config/local # mount config dir
name: user-config
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
Expand All @@ -187,27 +187,27 @@
type: Directory
name: root
- hostPath:
path: /etc/ilogtail-ilogtail-ds/checkpoint
path: /etc/loongcollector-loongcollector-ds/checkpoint
type: DirectoryOrCreate
name: checkpoint
- configMap:
defaultMode: 420
name: ilogtail-user-cm
name: loongcollector-user-cm
name: user-config
```

您也可以直接从下面的地址下载示例配置。

```bash
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-daemonset.yaml
wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/loongcollector-daemonset.yaml
```

添加参数示例

```yaml
- name: logtail
- name: loongcollector
command:
- /usr/local/ilogtail/ilogtail_control.sh
- /usr/local/loongcollector/loongcollector_control.sh
args:
- "start_and_block"
- "-enable_containerd_upper_dir_detect=true"
Expand All @@ -218,7 +218,7 @@
应用上述配置

```bash
kubectl apply -f ilogtail-daemonset.yaml
kubectl apply -f loongcollector-daemonset.yaml
```

4. 部署用来测试的nginx
Expand Down Expand Up @@ -271,19 +271,19 @@
5. 发送请求构造示例日志

```bash
kubectl exec nginx-<pod-id> -- curl localhost/hello/ilogtail
kubectl exec nginx-<pod-id> -- curl localhost/hello/loongcollector
```

6. 查看采集到的测试容器标准输出日志

```bash
kubectl logs ilogtail-ds-<pod-id> -n ilogtail
kubectl logs loongcollector-ds-<pod-id> -n loongcollector
```

结果为

```json
2022-07-14 16:36:50 {"_time_":"2022-07-15T00:36:48.489153485+08:00","_source_":"stdout","_image_name_":"docker.io/library/nginx:latest","_container_name_":"nginx","_pod_name_":"nginx-76d49876c7-r892w","_namespace_":"default","_pod_uid_":"07f75a79-da69-40ac-ae2b-77a632929cc6","_container_ip_":"10.223.0.154","remote_addr":"::1","remote_user":"-","time_local":"14/Jul/2022:16:36:48","method":"GET","url":"/hello/ilogtail","protocol":"HTTP/1.1","status":"404","body_bytes_sent":"153","http_referer":"-","http_user_agent":"curl/7.74.0","http_x_forwarded_for":"-","__time__":"1657816609"}
2022-07-14 16:36:50 {"_time_":"2022-07-15T00:36:48.489153485+08:00","_source_":"stdout","_image_name_":"docker.io/library/nginx:latest","_container_name_":"nginx","_pod_name_":"nginx-76d49876c7-r892w","_namespace_":"default","_pod_uid_":"07f75a79-da69-40ac-ae2b-77a632929cc6","_container_ip_":"10.223.0.154","remote_addr":"::1","remote_user":"-","time_local":"14/Jul/2022:16:36:48","method":"GET","url":"/hello/loongcollector","protocol":"HTTP/1.1","status":"404","body_bytes_sent":"153","http_referer":"-","http_user_agent":"curl/7.74.0","http_x_forwarded_for":"-","__time__":"1657816609"}
```

## 采集模版
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2022 iLogtail Authors
# Copyright 2022 loongcollector Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -15,23 +15,23 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ilogtail-ds
namespace: ilogtail
name: loongcollector-ds
namespace: loongcollector
labels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
spec:
selector:
matchLabels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
template:
metadata:
labels:
k8s-app: logtail-ds
k8s-app: loongcollector-ds
spec:
tolerations:
- operator: Exists # deploy on all nodes
containers:
- name: logtail
- name: loongcollector
env:
- name: ALIYUN_LOG_ENV_TAGS # add log tags from env
value: _node_name_|_node_ip_
Expand All @@ -45,24 +45,24 @@ spec:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: cpu_usage_limit # iLogtail's self monitor cpu limit
- name: cpu_usage_limit # loongcollector's self monitor cpu limit
value: "1"
- name: mem_usage_limit # iLogtail's self monitor mem limit
- name: mem_usage_limit # loongcollector's self monitor mem limit
value: "512"
- name: default_access_key_id # accesskey id if you want to flush to SLS
valueFrom:
secretKeyRef:
name: ilogtail-secret
name: loongcollector-secret
key: access_key_id
optional: true
- name: default_access_key # accesskey secret if you want to flush to SLS
valueFrom:
secretKeyRef:
name: ilogtail-secret
name: loongcollector-secret
key: access_key
optional: true
image: >-
sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:latest
sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/loongcollector-community-edition/loongcollector:latest
imagePullPolicy: IfNotPresent
resources:
limits:
Expand All @@ -78,16 +78,16 @@ spec:
mountPropagation: HostToContainer
name: root
readOnly: true
- mountPath: /usr/local/ilogtail/checkpoint # for checkpoint between container restart
- mountPath: /usr/local/loongcollector/data # for checkpoint between container restart
name: checkpoint
- mountPath: /usr/local/ilogtail/config/local # mount config dir
- mountPath: /usr/local/loongcollector/conf/continuous_pipeline_config/local # mount config dir
name: user-config
readOnly: true
lifecycle:
preStop:
exec:
command:
- /usr/local/ilogtail/ilogtail_control.sh
- /usr/local/loongcollector/loongcollector_control.sh
- stop
- "3"
livenessProbe:
Expand All @@ -112,10 +112,10 @@ spec:
type: Directory
name: root
- hostPath:
path: /etc/ilogtail-ilogtail-ds/checkpoint
path: /etc/loongcollector-loongcollector-ds/checkpoint
type: DirectoryOrCreate
name: checkpoint
- configMap:
defaultMode: 420
name: ilogtail-user-cm
name: loongcollector-user-cm
name: user-config
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2022 iLogtail Authors
# Copyright 2022 loongcollector Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -15,4 +15,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: ilogtail
name: loongcollector
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2022 iLogtail Authors
# Copyright 2022 loongcollector Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -15,8 +15,8 @@
apiVersion: v1
kind: Secret
metadata:
name: ilogtail-secret
namespace: ilogtail
name: loongcollector-secret
namespace: loongcollector
type: Opaque
data:
access_key_id: # accesskey id if you want to flush to SLS
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2022 iLogtail Authors
# Copyright 2022 loongcollector Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -15,8 +15,8 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: ilogtail-user-cm
namespace: ilogtail
name: loongcollector-user-cm
namespace: loongcollector
data:
nginx_stdout.yaml: |
enable: true
Expand Down
3 changes: 2 additions & 1 deletion scripts/dist.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,9 @@ cp "${ROOTDIR}/${OUT_DIR}/loongcollector" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}
cp "${ROOTDIR}/${OUT_DIR}/libGoPluginAdapter.so" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}"
cp "${ROOTDIR}/${OUT_DIR}/libGoPluginBase.so" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}"
mkdir -p "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/conf/instance_config/local/"
mkdir -p "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/conf/continuous_pipeline_config/local/"
cp "${ROOTDIR}/${OUT_DIR}/conf/instance_config/local/loongcollector_config.json" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/conf/instance_config/local/"
cp -a "${ROOTDIR}/${OUT_DIR}/conf/continuous_pipeline_config/local" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/conf"
cp -a "${ROOTDIR}/${OUT_DIR}/conf/continuous_pipeline_config/local" "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/conf/continuous_pipeline_config"
if file "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}/loongcollector" | grep x86-64; then ./scripts/download_ebpflib.sh "${ROOTDIR}/${DIST_DIR}/${PACKAGE_DIR}"; fi

# Splitting debug info at build time with -gsplit-dwarf does not work with current gcc version
Expand Down
Loading