I would like to understand better. It seems that OpenAI supports streaming + structured outputs.
If we are making this choice for simplicity reasons, I would be more specific: "The OpenAIChatGenerator does not ..."

Hmm interesting. I think because of the beta code example in documentation I misunderstood that its an unstable feature (not available in completions API). But you are right its supported. I’ll update the function to enable this.

• I did not notice that it was beta
• It might also be reasonable to skip it if it requires rewriting significantly the component. (Unfortunately, we have many other components depending on this implementation)
• If there are stable ways to do this, let's do it. Otherwise, let's create an issue to track this once that API is no longer in beta.

• I looked into their stream function and here looks like the stable version also supports response_format.
• I will test this locally and will update the code.

@anakin87 Looked into this and here are some points:

• The response_format with streaming will be passed to chat.completions.create. This endpoint allows response_format to be either a json schema or { "type": "json_object" }. But it cannot be a pydantic.BaseModel.
• From the documentation, it seems like the beta version supports pydantic models with stream endpoint. Which I dont want to introduce for the reasons you mentioned above.

So for now, I believe we can support the first point and mention the limitation in docstrings. Anyways, the error is handled by OpenAI itself if the user passes a pydantic model with streaming.

Agree... Let's do what you propose and mention in docstrings that Pydantic response_format won't work with streaming.

I find this hard to understand. Something similar to this would work?

We could always return a dictionary with the same fields from _prepare_api_call.

        if api_args.get("response_format"):
            # We cannot pass stream param to chat.completions.parse endpoint
            api_args.pop("stream", None)
            chat_completion = self.client.chat.completions.parse(**api_args)
      else:
            ...

(I might be wrong, in any case I'd appreciate it if we can make the code more intuitive to follow.)

Hmm we allow passing response_format with stream param to create endpoint, for streaming structured outputs so this wont work.

Ok I now understand, but it's hard to follow.

What I would recommend is to

1. include in api_args an item called endpoint/method containing "parse" or "create" (in _prepare_api_call)
2. (add comments where this value is set to explain why we are doing that.)
3. reuse the value in run

I think we should add an extra else here that raises an exception in case an unexpected value is popped? It would avoid all the typing issues below

I updated the condition to use parse if its passed as the endpoint, else always use create which was the case before.

Is there a different way to import this function that doesn't go through a private file? I'm a little worried the import path is subject to break/change

Hmm, for now seems like this is the only way to import this.

hmm I think we can do this a different way. We should be able to directly use the one from pydantic which looks like this parameters_schema = model.model_json_schema(). This is from the _create_tool_parameters_schema function in ComponentTool

OpenAI expect a stricter JSON Schema than Pydantic’s default. For example, objects must set additionalProperties and Optional keys are handled differently. As a result, model_json_schema() often isn’t accepted as-is.
Its also discussed here where another solution is offered but I prefer using the openai method over some unpopular library.

Nevertheless, I spotted a bug in to_dict where schema wasn't stored properly. Fixing this.

ahh okay thanks for the info