-
Notifications
You must be signed in to change notification settings - Fork 185
Labels
Accepting Pull RequestsLoggingLogging & Monitoring with MeltanoLogging & Monitoring with Meltanogood first issueGood for newcomersGood for newcomerskind/Featureuxvaluestream/Meltano
Description
Feature scope
CLI (options, error messages, logging, etc.)
Description
Right now, when both extractor and loader emit SDK structured logs, the output looks something like this:
2025-10-18T01:41:34.050193Z [info ] Environment 'dev' is active
2025-10-18T01:41:34.137423Z [info ] Running job with name 'dev:tap-fedidb-to-target-csv' and run ID '0199f4fa-b10e-7a29-bbe7-e819be8d2220'
2025-10-18T01:41:34.172081Z [info ] Using systemdb state backend
2025-10-18T01:41:34.172644Z [warning ] No state was found, complete import.
2025-10-18T01:41:34.308572Z [info ] tap-fedidb v0.5.0, Meltano SDK v0.50.2 name=tap-fedidb
2025-10-18T01:41:34.308905Z [info ] Skipping parse of env var settings... name=tap-fedidb
2025-10-18T01:41:34.309466Z [info ] Beginning full_table sync of 'popular_accounts' name=tap-fedidb
2025-10-18T01:41:34.785773Z [info ] METRIC metric_name=http_request_duration metric_value=0.468653 name=tap-fedidb
2025-10-18T01:41:34.798617Z [info ] METRIC metric_name=http_request_count metric_value=1 name=tap-fedidb
2025-10-18T01:41:34.798801Z [info ] METRIC metric_name=sync_duration metric_value=0.48907017707824707 name=tap-fedidb
2025-10-18T01:41:34.798890Z [info ] METRIC metric_name=record_count metric_value=136 name=tap-fedidb
2025-10-18T01:41:34.798951Z [info ] Skipping deselected stream 'servers'. name=tap-fedidb
2025-10-18T01:41:34.799008Z [info ] Beginning full_table sync of 'software' name=tap-fedidb
2025-10-18T01:41:35.014505Z [info ] METRIC metric_name=http_request_duration metric_value=0.214582 name=tap-fedidb
2025-10-18T01:41:35.023827Z [info ] METRIC metric_name=http_request_count metric_value=1 name=tap-fedidb
2025-10-18T01:41:35.023996Z [info ] METRIC metric_name=sync_duration metric_value=0.22496819496154785 name=tap-fedidb
2025-10-18T01:41:35.024072Z [info ] METRIC metric_name=record_count metric_value=63 name=tap-fedidb
2025-10-18T01:41:35.554208Z [info ] target-csv v0.2.1, Meltano SDK v0.50.2 name=target-csv
2025-10-18T01:41:35.554431Z [info ] Skipping parse of env var settings... name=target-csv
2025-10-18T01:41:35.558373Z [info ] Writing to destination file '/Users/edgarramirez/Code/meltano/triage/_console/popular_accounts.csv'... name=target-csv
2025-10-18T01:41:35.570079Z [info ] Writing to destination file '/Users/edgarramirez/Code/meltano/triage/_console/software.csv'... name=target-csv
2025-10-18T01:41:35.573459Z [info ] Reader 'target-csv' completed processing 203 lines of input (2 schemas, 199 records, 0 batch manifests, 2 state messages, 0 activate version messages). name=target-csv
2025-10-18T01:41:35.578880Z [info ] Appending 136 records to file... name=target-csv
2025-10-18T01:41:35.579005Z [info ] Appending 63 records to file... name=target-csv
2025-10-18T01:41:35.581862Z [info ] METRIC metric_name=batch_processing_time metric_value=0.0028040409088134766 name=target-csv
2025-10-18T01:41:35.582075Z [info ] METRIC metric_name=batch_processing_time metric_value=0.0033750534057617188 name=target-csv
2025-10-18T01:41:35.591689Z [info ] METRIC metric_name=record_count metric_value=136 name=target-csv
2025-10-18T01:41:35.591811Z [info ] METRIC metric_name=record_count metric_value=63 name=target-csv
2025-10-18T01:41:35.596854Z [info ] Using systemdb state backend
2025-10-18T01:41:35.598817Z [info ] Incremental state has been updated at 2025-10-18 01:41:35.598806+00:00.
2025-10-18T01:41:35.629481Z [info ] Block run completed
2025-10-18T01:41:35.629633Z [info ] Run completed
notice that the structured plugin log gets a name=<plugin name> that's displayed at the rightmost.
An arguable better experience would be for the plugin name (when available via structured logging) to be displayed in its own "column", similar to how the timestamp or the log level are displayed:
2025-10-18T01:41:34.050193Z [info ] meltano Environment 'dev' is active
2025-10-18T01:41:34.137423Z [info ] meltano Running job with name 'dev:tap-fedidb-to-target-csv' and run ID '0199f4fa-b10e-7a29-bbe7-e819be8d2220'
2025-10-18T01:41:34.172081Z [info ] meltano Using systemdb state backend
2025-10-18T01:41:34.172644Z [warning ] meltano No state was found, complete import.
2025-10-18T01:41:34.308572Z [info ] tap-fedidb v0.5.0, Meltano SDK v0.50.2
2025-10-18T01:41:34.309466Z [info ] tap-fedidb Beginning full_table sync of 'popular_accounts'
2025-10-18T01:41:34.798951Z [info ] tap-fedidb Skipping deselected stream 'servers'.
2025-10-18T01:41:34.799008Z [info ] tap-fedidb Beginning full_table sync of 'software'
2025-10-18T01:41:35.554208Z [info ] target-csv v0.2.1, Meltano SDK v0.50.2
2025-10-18T01:41:35.578880Z [info ] target-csv Appending 136 records to file...
2025-10-18T01:41:35.579005Z [info ] target-csv Appending 63 records to file...
2025-10-18T01:41:35.629633Z [info ] meltano Run completed
Fortunately, structlog supports this via passing custom Column instances to ConsoleRenderer.
Related
Metadata
Metadata
Assignees
Labels
Accepting Pull RequestsLoggingLogging & Monitoring with MeltanoLogging & Monitoring with Meltanogood first issueGood for newcomersGood for newcomerskind/Featureuxvaluestream/Meltano