Summary

Currently, the Okta source component does not function as expected. Vector never updates the since API query parameter to bring in a new window of Okta logs; the static since value means that the window only grows and grows with each interval. Additionally, because an until parameter is never provided, the Okta API always includes header links in the response instead of only on pagination. This leads to scenarios where Vector will quickly hit the Okta API rate limit.

This PR thus makes the following changes:

• Update since query every interval to bring in new logs.
• Remove dangery support for following header links.
• Rewrite the Okta implementation to look more like an HTTP Client source.
• Add Okta failure modes to the integration tests.

This is my first PR to Vector, and I'm very open to feedback! 😄

I associated this PR to an existing closed issue, where another PR started the Okta work. In building on top of that work, if it is more prudent to open a separate issue for what I'm seeing above, I'm happy to knock that out as well.

I did my best in trying to preserve the original intent with the since argument, but I wasn't able to find my way towards a better implementation with the structure of the code as-is. I'm open to suggestions as to how to make this component feel more "Vector-native"!

Furthermore, it is worth disclosing that no AI tools were used in the development of this PR.

Vector configuration

api:
  address: 127.0.0.1:8686
  enabled: false
  playground: false

data_dir: /tmp/vector-data-dir

sources:
  okta:
    type: okta
    domain: ***.okta.com
    token: ***
    scrape_interval_secs: 300
    decoding:
      codec: json

sinks:
  debug:
    inputs: [okta]
    type: "console"
    encoding:
      codec: json

How did you test this PR?

I built Vector and used the above configuration, alongside the VECTOR_LOG=debug flag, to diagnose and troubleshoot failures with the Okta source. For other components, I ran the test suite.

All changes have been human-tested with a release build of this branch against a live Okta environment.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Related: Okta system log source #22967

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.