-
Notifications
You must be signed in to change notification settings - Fork 3k
Description
Component(s)
extension/encoding/textencoding
What happened?
Description
The text_encoding
extension exposes configuration options for marshaling_separator
and unmarshaling_separator
. However, these settings do not appear to affect the exported output when used in combination with the fileexporter
.
On reviewing the implementation in extension/encoding/textencodingextension/extension.go
, it seems that the configured values for marshaling_separator
and unmarshaling_separator
are never applied when creating the textLogCodec
instance. As a result, the marshaling_separator
always defaults to an empty string, regardless of the user-provided configuration.
In addition, the current logic in MarshalLogs()
(extension/encoding/textencodingextension/text.go
) appends the separator after every log record. This behavior introduces trailing separators (e.g., empty lines when marshaling_separator="\n"
) instead of joining records in the same way as a standard delimiter would (e.g., "\n".join(records)
).
Steps to Reproduce
- Configure a
filelog
receiver to read logs from a file. - Configure a
fileexporter
withtext_encoding
and setmarshaling_separator
to"\n"
. - Observe the exported log file.
Expected Result
The following should be the expected log output written by fileexporter
.
<ip_address> - - [19/Sep/2025:10:04:06 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:06 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:06 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:06 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:06 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
Actual Result
- Configured
marshaling_separator
value is ignored resulting in log lines within aResourceLog
getting concatenated together.
<ip_address> - - [19/Sep/2025:10:04:06 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"<ip_address> - - [19/Sep/2025:10:04:06 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:06 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:06 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"<ip_address> - - [19/Sep/2025:10:04:06 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
- When configured properly, the separator is appended after each record, resulting in extra blank lines in the output when using newline separator for marshaling.
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
<ip_address> - - [19/Sep/2025:10:04:07 +0000] "POST /api/v1/service/log HTTP/1.1" 200 13 "-" "LuaSocket 3.0.0"
...
Collector version
0.135.0
Environment information
Environment
OS: Debian 11
OpenTelemetry Collector configuration
# Sample Otelcol configs
receivers:
filelog:
include:
- /var/log/nginx/access.log
start_at: end
include_file_path: true
include_file_name: false
resource:
environment: development
component: nginx
operators:
- type: move
from: attributes["log.file.path"]
to: resource["path"]
- type: add
field: resource.log_path
value: EXPR(join([resource.environment,resource.component], "-") + resource.path)
processors:
batch:
send_batch_size: 8192
timeout: 200ms
send_batch_max_size: 0
memory_limiter:
check_interval: 5s
limit_percentage: 75
spike_limit_percentage: 10
exporters:
file:
path: /var/log/otelcol/*.log
rotation:
max_megabytes: 5000
max_days: 7
max_backups: 4
localtime: false
encoding: text_encoding/utf8
group_by:
enabled: true
resource_attribute: log_path
extensions:
text_encoding/utf8:
encoding: utf8
marshaling_separator: "\n"
service:
extensions: [text_encoding/utf8]
pipelines:
logs:
receivers: [filelog]
processors: [memory_limiter,batch]
exporters: [file]
Log output
Additional context
Only checked the MarshalLogs()
logic. The logic used in UnmarshalLogs()
was not checked for correctness.
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1
or me too
, to help us triage it. Learn more here.