Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@lmarques03
Copy link
Contributor

Occasionally when retrieving the dashboard information, an error would occur with the message:
"msg="status: 404, body: {\"message\":\"Dashboard not found\"}"

After some investigation it was concluded that the error occurred when the request to Grafana API coincided with the restart of Grafana pods.
This PR adds a one-time retry if the GET request fails.

@nicolastakashi
Copy link
Owner

Hey, @lmarques03 thanks for the contribution!
Can you elaborate a little bit more about this issue, looking at the description that you provided, seems to me that the dashboard doesn't exist, could it be an issue with the dashboard sync instead of the Grafana API?

@lmarques03
Copy link
Contributor Author

Hello, @nicolastakashi.
It certainly could be, however from our side, every time the error was triggered it coincided with the restart of Grafana pods. As such, the error is only triggered very occasionally, which indicates that most of the times the dashboard sync is done correctly. Also, the error does not persist, and on the next API call everything works fine, and the dashboard is found.

Apart from all of this, I was not able to replicate the error in my local environment, even when importing the same dashboards as in prd.

@nicolastakashi
Copy link
Owner

Hey, @lmarques03 thanks for the clarification.
Well since it's a transient issue in my view we can just check the error code and don't treat 404 as an error, this is a recurrent task, the next time the dashboards will be available and everything is ok.

WDT?

@lmarques03
Copy link
Contributor Author

@nicolastakashi That sounds like a good idea. Would you prefer to mantain the retry (and do it only when the error is 404), or simply ignore the error altogether, and wait for the next iteration?

@nicolastakashi
Copy link
Owner

Hi @lmarques03 I would say we can just handle the 404 as an warn instead of an error, because the retry is out of the box implemented by the pooling interval, wdyt?

@lmarques03
Copy link
Contributor Author

Hello @nicolastakashi. I changed the code to handle 404 as a warning. It should be fine now. Please review and let me know what you think.

Copy link
Owner

@nicolastakashi nicolastakashi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you're using continue you don't need the else on the if statement

@lmarques03
Copy link
Contributor Author

Yeah that was dumb. I fixed it now.
If the dashboard retrieval results in an error, then it always continues to the next iteration. But it does not consider the 404 response as an error.

@nicolastakashi nicolastakashi merged commit 0f7ea5b into nicolastakashi:main Jan 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants