Allow even-parentless workflow spans to always be created #817

cretz · 2025-04-07T13:49:11Z

What was changed

Added always_create_workflow_spans bool to OTel TracingInterceptor constructor. Docs best describe the situation:

            always_create_workflow_spans: When false, the default, spans are
                only created in workflows when an overarching span from the
                client is present. In cases of starting a workflow elsewhere,
                e.g. CLI or schedules, a client-created span is not present and
                workflow spans will not be created. Setting this to true will
                create spans in workflows no matter what, but there is a risk of
                them being orphans since they may not have a parent span after
                replaying.

Checklist

Closes [Feature Request] Make option for OTel workflow spans even if client span not present #794

Fixes temporalio#794

yuandrew · 2025-04-09T16:59:30Z

temporalio/contrib/opentelemetry.py

                name=span_name,
                # Always set span attributes as workflow ID and run ID
                attributes=attributes,
                time_ns=temporalio.workflow.time_ns(),
                link_context=link_context_carrier,
                exception=exception,
                kind=kind,
+                parent_missing=opentelemetry.trace.get_current_span()
+                is opentelemetry.trace.INVALID_SPAN,


nit: I'm not familiar enough with python style, should the 2nd line here be indented to indicate these 2 lines are actually 1?

This is what the auto formatter did

We should place parens around expressions in this situation

yuandrew · 2025-04-09T17:00:43Z

tests/contrib/test_opentelemetry.py

+            id=f"workflow_{uuid.uuid4()}",
+            task_queue=worker.task_queue,
+        )
+    # Confirm the spans are not there


Suggested change

# Confirm the spans are not there

# Confirm the spans are there

dandavison · 2025-04-09T20:56:49Z

temporalio/contrib/opentelemetry.py

+                parent_missing=opentelemetry.trace.get_current_span()
+                is opentelemetry.trace.INVALID_SPAN,


Suggested change

parent_missing=opentelemetry.trace.get_current_span()

is opentelemetry.trace.INVALID_SPAN,

parent_missing=(

opentelemetry.trace.get_current_span()

is opentelemetry.trace.INVALID_SPAN

),

dandavison · 2025-04-09T21:07:15Z

temporalio/contrib/opentelemetry.py

+        # If the parent is missing and user hasn't said to always create, do not
+        # create
+        if params.parent_missing and not self._always_create_workflow_spans:
+            return None


This line looks like a breaking change, am I getting that wrong?

This logic was inside the workflow's form of _completed_span before as:

# If there is no span on the context, we do not create a span if opentelemetry.trace.get_current_span() is opentelemetry.trace.INVALID_SPAN: return None

but now that I have to check a parameter from outside the sandbox, I moved the logic to the outside-of-sandbox part instead of the inside-of-sandbox part.

dandavison · 2025-04-09T21:13:05Z

temporalio/contrib/opentelemetry.py

+                workflow spans will not be created. Setting this to true will
+                create spans in workflows no matter what, but there is a risk of
+                them being orphans since they may not have a parent span after
+                replaying.


I'm not swapped into this work.

Why don't we default to always creating the parent-less spans? Wouldn't that be more useful to users than dropping them?

The docstring here uses the term "orphan" but couldn't they equally be viewed as roots, originating in the workflow?

[Just a question, not blocking this PR] Could it make sense to allow tracing to be enabled in the CLI (and maybe even Schedule starter one day) when starting workflows?

Why don't we default to always creating the parent-less spans? Wouldn't that be more useful to users than dropping them?

Why we don't now - because it was chosen not to originally and we can't just change on people. Why we didn't originally - because if you create spans without a parent you have orphans. So if it was cached (i.e. never replayed), it'd just be under RunWorkflow which is created on non-replay start, but when it is replayed, everything after has no parent so it is on its own.

The docstring here uses the term "orphan" but couldn't they equally be viewed as roots, originating in the workflow?

It means spans like StartActivity may or may not have a parent, depending on whether the workflow is running somewhere separate than when it first created the RunWorkflow span. People do not expect spans from inside a workflow to be without a parent in my experience.

Could it make sense to allow tracing to be enabled in the CLI (and maybe even Schedule starter one day) when starting workflows?

Yes it can, though OTel usually expects people to programmatically configure tracers, not outside of code. But CLI could definitely accept everything it needs to build https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry#NewTracingInterceptor (basically it'd be whatever was required to build a Go tracer).

To clarify what's happening here: client-side start workflow creates StartWorkflow, then first non-replay start creates RunWorkflow and sets that on context (if there's the StartWorkflow parent), then execute activity creates StartActivity (implicitly parenting to RunWorkflow if it's in this instance, StartWorkflow otherwise). So StartWorkflow is the only stable span available. There has been talk of temporalio/features#394 to help the situation where a span was not created by the starter, but in the meantime default Python (unlike some other SDKs) chose not to potentially create orphans by default. This option allows orphans to happen. I hope that's clear.

Thanks, yes that's very helpful.

Why don't we default to always creating the parent-less spans? Wouldn't that be more useful to users than dropping them?

Why we don't now - because it was chosen not to originally and we can't just change on people.

Would that really be a (bad) breaking change? Wouldn't it just mean some new traces show up in their observability platform that didn't before?

Would that really be a (bad) breaking change?

Yes, I think it'd be a bad breaking change. I also think the default that exists is valuable even if we were ok with breaking changes. Orphaned spans not under a parent can cause those looking at traces for a workflow to not see a span.

Wouldn't it just mean some new traces show up in their observability platform that didn't before?

Yes, which can clutter a tracing platform. Today people can trust that they're not just going to have some StartActivity top-level span flood the top-level of their Jaeger list.

Allow even-parentless workflow spans to always be created

54c33e7

Fixes temporalio#794

cretz requested a review from a team as a code owner April 7, 2025 13:49

yuandrew reviewed Apr 9, 2025

View reviewed changes

yuandrew approved these changes Apr 9, 2025

View reviewed changes

dandavison reviewed Apr 9, 2025

View reviewed changes

dandavison approved these changes Apr 9, 2025

View reviewed changes

cretz added 2 commits April 10, 2025 14:29

Merge branch 'main' into otel-always-workflow-span

6452db2

Merge branch 'main' into otel-always-workflow-span

0e83229

cretz merged commit 1296cd7 into temporalio:main Apr 10, 2025
14 checks passed

cretz deleted the otel-always-workflow-span branch April 10, 2025 18:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow even-parentless workflow spans to always be created #817

Allow even-parentless workflow spans to always be created #817

cretz commented Apr 7, 2025

yuandrew Apr 9, 2025

cretz Apr 9, 2025

dandavison Apr 9, 2025

yuandrew Apr 9, 2025

dandavison Apr 9, 2025

dandavison Apr 9, 2025

cretz Apr 9, 2025 •

edited

Loading

dandavison Apr 9, 2025 •

edited

Loading

cretz Apr 9, 2025 •

edited by dandavison

Loading

dandavison Apr 9, 2025

cretz Apr 10, 2025

	# Confirm the spans are not there
	# Confirm the spans are there

		parent_missing=opentelemetry.trace.get_current_span()
		is opentelemetry.trace.INVALID_SPAN,

Allow even-parentless workflow spans to always be created #817

Allow even-parentless workflow spans to always be created #817

Conversation

cretz commented Apr 7, 2025

What was changed

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cretz Apr 9, 2025 • edited Loading

Choose a reason for hiding this comment

dandavison Apr 9, 2025 • edited Loading

Choose a reason for hiding this comment

cretz Apr 9, 2025 • edited by dandavison Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cretz Apr 9, 2025 •

edited

Loading

dandavison Apr 9, 2025 •

edited

Loading

cretz Apr 9, 2025 •

edited by dandavison

Loading