Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Auto instrumentation is not working #3482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gyliu513 opened this issue Oct 25, 2023 · 19 comments
Open

Auto instrumentation is not working #3482

gyliu513 opened this issue Oct 25, 2023 · 19 comments
Labels
auto-instrumentation related to auto-instrumentation of the sdk bug Something isn't working

Comments

@gyliu513
Copy link
Member

Following the steps here https://opentelemetry.io/docs/instrumentation/python/automatic/example/#execute-the-automatically-instrumented-server

Run server first

opentelemetry-instrument --traces_exporter console --metrics_exporter none python server_automatic.py

Then run client

python client.py testing
{
    "name": "client-server",
    "context": {
        "trace_id": "0x8212674bb3c491c6ca3995ba997d90bc",
        "span_id": "0xe8254224de6581ba",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0x71f65d5cce1f1f63",
    "start_time": "2023-10-25T17:24:31.980542Z",
    "end_time": "2023-10-25T17:24:31.992175Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.20.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}
{
    "name": "client",
    "context": {
        "trace_id": "0x8212674bb3c491c6ca3995ba997d90bc",
        "span_id": "0x71f65d5cce1f1f63",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2023-10-25T17:24:31.980484Z",
    "end_time": "2023-10-25T17:24:31.992201Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.20.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}

Client do have output, but server automatic do not have any output, based on the doc here https://opentelemetry.io/docs/instrumentation/python/automatic/example/#execute-the-automatically-instrumented-server

The console running server_automatic.py will display the spans generated by instrumentation as JSON. The spans should appear similar to the following example:
@gyliu513 gyliu513 added the bug Something isn't working label Oct 25, 2023
@kquinsland
Copy link

kquinsland commented Nov 13, 2023

I have hit the same issue while trying to debug auto-instrumentation with an internal non-flask app.

Even when explicitly setting arguments via the CLI, I get nothing in either the console or the debug logs of the otel-collector that I have running locally:

❯ python3 --version
Python 3.12.0
❯ opentelemetry-instrument \
    --traces_exporter console,otlp \
    --metrics_exporter console,otlp \
    --service_name your-service-name \
    --exporter_otlp_endpoint 127.0.0.1:4317 \
    --exporter_otlp_insecure=true \
    python3 server_automatic.py
 * Tip: There are .env or .flaskenv files present. Do "pip install python-dotenv" to use them.
 * Serving Flask app 'server_automatic'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:8082
Press CTRL+C to quit
testing
127.0.0.1 - - [13/Nov/2023 12:43:45] "GET /server_request?param=testing HTTP/1.1" 200 -

I do see json dumped to the console when using the client.py though.


I was wondering if the 3.12 was breaking things and changed venv to 3.11.6 and still nothing:

❯ python3 --version
Python 3.11.6
❯ opentelemetry-instrument --version
opentelemetry-instrument 0.42b0
❯ opentelemetry-instrument \
    --traces_exporter console,otlp \
    --metrics_exporter console,otlp \
    --service_name your-service-name \
    --exporter_otlp_endpoint 127.0.0.1:4317 \
    --exporter_otlp_insecure=true \
    python3 server_automatic.py
 * Tip: There are .env or .flaskenv files present. Do "pip install python-dotenv" to use them.
 * Serving Flask app 'server_automatic'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:8082
Press CTRL+C to quit
testing
127.0.0.1 - - [13/Nov/2023 13:00:34] "GET /server_request?param=testing HTTP/1.1" 200 -

@pcwiese
Copy link

pcwiese commented Nov 14, 2023

+1. The other 2 samples work just fine, but I get nothing on the server console when hitting server_request using automatic instrumentation.

@h1t35h
Copy link

h1t35h commented Nov 20, 2023

+1 Seems something have changed auto instrumentation isn't working for me as well. Even FastAPI auto instrumentation is broken.

@keisukesakasai
Copy link

I have hit the same issue with opentelemetry-instrumentation 0.42b0.

Will it work using 0.41b0?

@lzchen lzchen added the auto-instrumentation related to auto-instrumentation of the sdk label Nov 22, 2023
@panys9
Copy link

panys9 commented Nov 29, 2023

I can only make it work with python 3.7

@flands
Copy link
Contributor

flands commented Jan 3, 2024

The issue appears to be that Flask/Werkzeug 3.0+ is not supported by the auto instrumentation. If you install an older version, then it works as expected: pip uninstall Flask Werkzeug; pip install "Flask <3" "Werkzeug <3"

@flands
Copy link
Contributor

flands commented Jan 4, 2024

Directions can be updated to reflect current state until open-telemetry/opentelemetry-python-contrib#1975 is resolved.

@gsakkis
Copy link

gsakkis commented Feb 8, 2024

It's not just Flask, I can't get any ASGI framework auto instrumentation to work; I tried FastAPI, Starlette, plain ASGI.

FastAPI example:

# app.py

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
def read_root():
    return {"Hello": "World"}

Run server: opentelemetry-instrument --traces_exporter console --metrics_exporter none uvicorn app:app
Run client: curl 'localhost:8000/'
Installed packages:

$ pip freeze | grep opentelemetry-instrumentation

opentelemetry-instrumentation==0.43b0
opentelemetry-instrumentation-asgi==0.43b0
opentelemetry-instrumentation-fastapi==0.43b0
opentelemetry-instrumentation-flask==0.43b0
opentelemetry-instrumentation-httpx==0.43b0
opentelemetry-instrumentation-starlette==0.43b0
opentelemetry-instrumentation-wsgi==0.43b0

@aeb-dev
Copy link

aeb-dev commented Feb 13, 2024

Is there any update on this. We are having the same issue as @gsakkis

@gsakkis
Copy link

gsakkis commented Feb 14, 2024

@aeb-dev apparently auto instrumentation needs the opentelemetry-distro package as well, I opened a separate issue about it.

@aeb-dev
Copy link

aeb-dev commented Feb 14, 2024

I actually have opentelemetry-distro and it still does not work

@fromagge
Copy link

I was facing this even with the minimal example and reading the documentation is realized that I had missed this:

https://opentelemetry.io/docs/languages/python/automatic/example/#instrumentation-while-debugging

Basically on Flask when using debug=True you have to use use_reloader=False, if not the instrumentation may not worked.

This worked for me, I overlooked it.

@ijkbytes
Copy link

ijkbytes commented Jul 30, 2024

I actually have opentelemetry-distro and it still does not work

@aeb-dev
Did you finally find the cause of this issue? I'm also troubled in this.

@emdneto
Copy link
Member

emdneto commented Jul 30, 2024

@gyliu513 Gentle remind. Any reason to keep the issue open?

@aeb-kb01
Copy link

@ijkbytes
It happens when you use an asgi server with multiple processes

@DrLuke
Copy link

DrLuke commented Aug 9, 2024

If you use uvicorn with reloading like uvicorn.run("myapp:app", host="localhost", port=8000, reload=True) it will also not work. Set reload to False.

@karlmutch-cynch
Copy link

emdneto I think this is still occurring

@nallamr
Copy link

nallamr commented Feb 28, 2025

We are getting system metrics only if we use reload=true. If we turnoff or remove it works.
Version : 0.48b0

@gcavalcante8808
Copy link

@aeb-dev apparently auto instrumentation needs the opentelemetry-distro package as well, I opened a separate issue about it.

I get by this requirement two years ago and testing again today, opentelemetry-distro is really by opentelemetry-instrument to autoload related instrument libraries. Here are the libraries that I'm using:

opentelemetry-api==1.32.1
opentelemetry-distro==0.53b1
opentelemetry-exporter-otlp==1.32.1
opentelemetry-exporter-otlp-proto-common==1.32.1
opentelemetry-exporter-otlp-proto-grpc==1.32.1
opentelemetry-exporter-otlp-proto-http==1.32.1
opentelemetry-exporter-richconsole==0.53b1
opentelemetry-instrumentation==0.53b1
opentelemetry-instrumentation-aiohttp-client==0.53b1
opentelemetry-instrumentation-aiohttp-server==0.53b1
opentelemetry-instrumentation-asyncio==0.53b1
opentelemetry-instrumentation-click==0.53b1
opentelemetry-instrumentation-dbapi==0.53b1
opentelemetry-instrumentation-grpc==0.53b1
opentelemetry-instrumentation-httpx==0.53b1
opentelemetry-instrumentation-logging==0.53b1
opentelemetry-instrumentation-openai-v2==2.1b0
opentelemetry-instrumentation-psycopg2==0.53b1
opentelemetry-instrumentation-requests==0.53b1
opentelemetry-instrumentation-sqlalchemy==0.53b1
opentelemetry-instrumentation-sqlite3==0.53b1
opentelemetry-instrumentation-threading==0.53b1
opentelemetry-instrumentation-tortoiseorm==0.53b1
opentelemetry-instrumentation-urllib==0.53b1
opentelemetry-instrumentation-urllib3==0.53b1
opentelemetry-instrumentation-wsgi==0.53b1
opentelemetry-proto==1.32.1
opentelemetry-sdk==1.32.1
opentelemetry-semantic-conventions==0.53b1
opentelemetry-util-http==0.53b1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-instrumentation related to auto-instrumentation of the sdk bug Something isn't working
Projects
None yet
Development

No branches or pull requests