A thing worth discussing: it would be entirely possible to rescue the OSError and just return a boolean to indicate whether starting the listener succeeded. Given that the previous behaviour was to silently not serve metrics for the current process (instead of halting with an exception like this introduces), a smaller change would be to add a rescue here - and existing setups with duplicate metrics ports would still fail silently.

Thoughts?

There's also been requests to be able to stop the httpserver, so it'd be best to consider that with this change.

Hi @brian-brazil,

As I see it, one option would be to return the handle of the daemon thread from the start_http_server method, or save it off somewhere for future reference. Would that be a handy enough API?

Alternatively, the return value could be the thread-handle on successful service start, None on startup failure. What do you think?

I see, you probably refer to #76 - but as I read that issue, the user ended up finding a different way to start/stop their listener?

@brian-brazil fwiw, I've created code in my project to just reuse the prometheus_client.MetricsHandler to setup my own HTTP listener just like I want it (with automatic probing of a number of ports until a free one is found).

Thank you for making the prometheus_client.MetricsHandler reusable externally, I can only imagine how many variations people need for running their services.

Feel free to close this pull-request if the design is not something you can use going forward.

The PR as-is is good idea, the question is more around rehandling exceptions. The Pythonic way is to just let them be thrown.

(with automatic probing of a number of ports until a free one is found).

That smells a bit, your config management should be telling you what port number to use.

Thank you for making the prometheus_client.MetricsHandler reusable externally, I can only imagine how many variations people need for running their services.

That's the idea.

I concur with the port probing being potentially problematic, but given that a uwsgi master process spawns a number of child processes at various times, it seems to make sense they each pick a port in a known range where they make metrics available. I'm using the Telegraf daemon to connect to each of these ports and collect the available metrics. There is a simplicity to this I like - and since to the best of my knowledge each uwsgi worker-process should not care too much about its individual identity/number in the flock, it also makes sense that it cannot have a distinct static prometheus listener port.

So, any specific changes you'd like to see to this code at this time?

Sounds like you should be reading #66, that's not a safe way to do multi-process.

Thank you for your suggestion.

In #66 I see a way of having workers report their metrics up the process tree to the master process, but it is currently a work in progress. I do not see any discussion of safety properties that preclude individual workers from exposing their own metrics directly, but it may very well be too implicit for me to see it. Is it related to the point in the startup process where workers are spawned, that some internal structure may or may not be correctly setup yet?

If this is veering off-topic I do apologise.

The issue is that there might be state in a dead worker that you want to preserve past its demise, and that correctly collapsing per-process counters requires data from dead workers too.

Agreed, that would indeed be loss of recent state in this design. And since the uwsgi master process will much too happily kill -9 workers that are past their prime - dataloss will be guaranteed in that case if the collector did not sweep in soon enough.

To me that comes to the trade off of sending metrics somewhere external immediately while handling an incoming request (to guarantee survivability of metrics, but adding latency), or to buffer up measurements within each worker-process for a configurable short time, and gather the values in bulk every X seconds. All in-memory queueing means the potential for dataloss, but at much greater throughput. Adding additional communication complexity to the master/worker hierarchy as #66 does seems like it might be a good tradeoff, but things do become more difficult to reason about.

I'll close this merge-request now, as I think I have a grasp of the tradeoffs I'm making, and I think you are right the solution provided here is not generally applicable.