Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Slow streamers might skip files if it falls more than lookBackInterval behind the current folder #200

@jeppe742

Description

@jeppe742

We had an example yesterday were a stream received unusually large amount of data

https://app.datadoghq.eu/dashboard/xnk-i3d-xj7/arcane-20-streams?fromUser=true&fullscreen_end_ts=1758801720027&fullscreen_paused=false&fullscreen_refresh_mode=sliding&fullscreen_section=overview&fullscreen_start_ts=1758628920027&fullscreen_widget=8823882781046968&refresh_mode=sliding&tpl_var_StreamId%5B0%5D=mss-eu-general-journal-account-entry&from_ts=1758626671745&to_ts=1758799471745&live=true

Image

The stream was able to process the data, slowly merging the data into bronze, but eventually after several hours it for some reason decided to jump to current time - lookBackInterval which effective skipped all the files between 14:00-14:45

https://app.datadoghq.eu/logs?query=%40host%3Amss-eu-general-journal-account-entry-%2A%20%28%40messageTemplate%3A%22Reading%20file%20%25s%2F%25s%20from%20Azure%20Storage%20account%20%25s%22%29%20host%3Amss-eu-general-journal-account-entry-vcs4h&agg_m=count&agg_m_source=base&agg_q=host&agg_q_source=base&agg_t=count&clustering_pattern_field_path=message&cols=host%2Cservice%2C%23file&data_extractions=%5B%5B%22message%22%2C%22Reading%20file%20dataverse-prdeccocceu-unq732376bb88cd48fe9c00b159d4492%2F%25%7Bdata%3Afile%7DZ%2Fmodel.json%20from%20Azure%20Storage%20account%20daieu%22%5D%5D&event=AwAAAZl8oaY9vNrkIwAAABhBWmw4b2FhZkFBQWVBUHVCSVlIZjd3QUIAAAAkZjE5OTdjYWEtZWE2OC00ZTlhLTk1NjYtMmVhYmVkMzU2MGJkAABQoQ&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=10&top_o=top&viz=stream&x_missing=true&from_ts=1758718800000&to_ts=1758734460000&live=false

Image

Does it make sense that the stream jumps, even if there are still unprocessed files ?

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions