Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Log request order#1237

Draft
azplanlos wants to merge 1 commit intomicrocks:1.10.xfrom
azplanlos:log_request_order
Draft

Log request order#1237
azplanlos wants to merge 1 commit intomicrocks:1.10.xfrom
azplanlos:log_request_order

Conversation

@azplanlos
Copy link
Contributor

Description

This is a draft of a new functionality I want to suggest. It basically does two things:

  • create a ID to identify requests in logs based on a templated string in operation metadata (REST only currently)
  • create a log of invocations and save them to the database including timestamp information
  • provide a new REST endpoint to access the (n) newest invocations per service

If this contribution is an interesting improvement to microcks in general I would try to get it out of draft status and also add some unit tests to check the functionality. And maybe we could look into on how this could be utilized for other controller types as well. Currently we only need this for the REST controller and didn't look into other things.

Related issue(s)

@azplanlos azplanlos force-pushed the log_request_order branch from 63670f2 to 7799dda Compare July 12, 2024 07:37
@lbroudoux
Copy link
Member

Hey @azplanlos, thanks for proposing this feature.

It looks like you started your work from an outdated branch and pulled already merged commits. Could you please start from a freshly updated one to avoid noise of older commits and conflicts resolution?

Also, could you please:

  • provide some context on this feature: why it enables? what it the use-case and the capabilities it provides?
  • sign-off the commits otherwise the DCO checks will fail and we'll not be able to merge your work.

Than you!

@azplanlos
Copy link
Contributor Author

azplanlos commented Jul 12, 2024

Sure, I'll take care of that. This is still a draft and I'll rebase my work on the 1.10.x branch.

To give some more context about this feature:
We are currently setting up tests for testing how concurrent requests are propagated over multiple services. In particular one of our services under test is running as a facade in front of another service and we need to get information about the order these requests are received by the consuming service this facade is forwarding them after applying some logic. The implemented REST endpoint /api/log/invocations/xxx/0.0.0 gives us this information based on log entries created in the database. This is basically like inserting a probe in the mock. Additionally an ID is generated to uniquely identify the received requests and the format of this id can be specified within the microcks extension of the OpenAPI specification on operation level.

@lbroudoux please let me know if this is an interesting feature and I would add some more unit tests and documentation for this.

@lbroudoux
Copy link
Member

This looks like an interesting feature! Have you think of integration with existing trace systems like the one provided by OpenTelemetry/OpenTracing?

@azplanlos
Copy link
Contributor Author

You are totally right, including some more information in tracing totally makes sense too. I'll add the information needed to identify the requests to open telemetry's span information. But in order to conduct automatic testing we will need both ways.

@lbroudoux
Copy link
Member

Hey @azplanlos,
I'm back from some vacation and ready to review! Let me know when things are close to be stable (even the PR is still Draft) so that I can start having a look at it.
Cheers,

@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 30 days if no further activity occurs. To unstale this pull request, add a comment with detailed explanation.

There can be many reasons why some specific pull request has no activity. The most probable cause is lack of time, not lack of interest. Microcks is a Cloud Native Computing Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this pull request forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

@github-actions github-actions bot added the stale State due to inactivity label Aug 22, 2024
@lbroudoux lbroudoux added keep-open Explicitily keep open and removed stale State due to inactivity labels Aug 26, 2024
@lbroudoux
Copy link
Member

Hey @azplanlos,
Any news on this topic? This is still marked as Draft but I definitely think it's an interesting feature.
Need any help on this?

@azplanlos
Copy link
Contributor Author

azplanlos commented Aug 26, 2024 via email

@lbroudoux
Copy link
Member

Hey @azplanlos,
Any news on this topic?

@getpantoai
Copy link

PR Summary:

This PR enhances logging by tracking and persisting invocation orders for mock requests. Key changes include adding a new InvocationLogEntry model, repositories, and a listener (InvocationAdvancedMetrics) to log and retrieve invocation events, along with updates to various controllers and event constructors to support request IDs and proper ordering of log entries.

Reviewed by Panto AI


@Override
public void onApplicationEvent(@NonNull MockInvocationEvent event) {
repo.insert(new InvocationLogEntry(event.getTimestamp(), event.getServiceName(), event.getServiceVersion(),

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace event.getTimestamp() with event.getInvocationTimestamp().getTime() to ensure a valid epoch value is passed to the log entry.

repo.insert(new InvocationLogEntry(
    event.getInvocationTimestamp().getTime(),
    event.getServiceName(),
    event.getServiceVersion(),
    event.getMockResponse(),
    event.getInvocationTimestamp(),
    event.getDuration(),
    event.getSource().getClass().getSimpleName(),
    event.getRequestId()
));

@getpantoai
Copy link

Reviewed up to commit:c98b8ce5656774573a7484fb7a4d0394953b7c72

Additional Suggestion
Others - Consider unifying id generation by using invocationTimestamp consistently and remove or clarify the purpose of timestampEpoch field. - Add a getter for the invocation timestamp or update consumers (e.g. InvocationAdvancedMetrics) to use getInvocationTimestamp() for obtaining the epoch millis.

Reviewed by Panto AI


Few more points:

  • Remove duplicate definitions for mocks.enable-invocation-stats and CORS properties to avoid configuration conflicts.

Copy link
Member

@lbroudoux lbroudoux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there, thanks a lot for the contributIon! A bot has make this one revival to my mind and I wanted to take the time to have a full review. As discussed in the past, this seems to be a super valuable use-case!

I've got some concerns regarding the persistence layer and I would love to have more explanations about the requestIdpart. I think having a live example with detailed use-case explanation could be nice as well. And I'll probably also request help for writing the documentation on that part too 😉

Also - and sorry because it seems to be a lot - could you rebase your branch on the 1.12.x branch from Microcks? This is where the devs happen at the moment.


import java.util.Date;

@Document("invocations")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this one is necessary. I prefer to limit the number of dependencies to spring-data stuff to the minimum.

Comment on lines +28 to +30
/**
*
*/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can remove unnecessary comments.


private Set<String> resourcePaths;
private List<ParameterConstraint> parameterConstraints;
private String idPath;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is actually the idPath?

/**
* Repository for InvocationLogEntries
*/
@NoRepositoryBean
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This construct and way to define repository is different than the one we used for DailyStatisticRepository. I find it a bit more difficult to understand - especially the InvocationLogRepositoryImpl part with the MongoEntityInformation override.

I think it could be nice to refactor reusing the same pattern here. What do you think?

private MongoTemplate mongoTemplate;

public InvocationLogRepositoryImpl(MongoOperations mongoOperations) {
super(new MongoEntityInformation<>() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment on interface definition.

* @param limit maximum number of latest results
* @return list of results ordered by newest entry first
*/
@GetMapping(value = "/log/invocations/{service}/{version}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We follow the plural form naming convention, so it should be /logs/invocations/{service}/{version}

services.update.interval=${SERVICES_UPDATE_INTERVAL:0 0 0/2 * * *}

mocks.enable-invocation-stats=${ENABLE_INVOCATION_STATS:true}
mocks.enable-invocation-logs=${ENABLE_INVOCATION_LOGS:true}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to disable this by default and make it explicitly opted-in. At least during the first releases, to be sure we don't have regression issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants