Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Backport #9888 to 4.19: Fix Usage inconsistencies #10712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: 4.19
Choose a base branch
from
Open

Backport #9888 to 4.19: Fix Usage inconsistencies #10712

wants to merge 4 commits into from

Conversation

winterhazel
Copy link
Member

Description

This PR backports #9888 to 4.19, fixing the issue described in #10687 and some other ones.

  • Duplicated entries in the networks helper table, resulting in duplicated usage record generation;
  • Port forwarding helper entries are never removed, resulting in removed port forwarding rules generating usage records;
  • Load balancer helper entries are never removed, resulting in removed load balancer rules generating usage records;
  • VM snapshot helper entries are never removed, resulting in removed VM snapshots generating usage records;
  • Helper entries for volumes in the secondary storage are never removed after the volume gets attached to a VM, resulting in secondary storage usage records being generated for volumes that are not in the secondary storage anymore.

I still intend to work on a normalization script before closing #10687, but it can be merged separately from this PR.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

I reproduced the tests described in #9888.

@winterhazel
Copy link
Member Author

@blueorangutan package

@winterhazel winterhazel added this to the 4.19.3 milestone Apr 14, 2025
@codecov Codecov
Copy link

codecov bot commented Apr 14, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 0% with 93 lines in your changes missing coverage. Please review.

Project coverage is 15.16%. Comparing base (f13cf59) to head (4534d7b).
Report is 22 commits behind head on 4.19.

Files with missing lines Patch % Lines
...rc/main/java/com/cloud/usage/UsageManagerImpl.java 0.00% 65 Missing ⚠️
...java/com/cloud/usage/dao/UsageNetworksDaoImpl.java 0.00% 14 Missing ⚠️
...main/java/com/cloud/usage/dao/UsageVpcDaoImpl.java 0.00% 14 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               4.19   #10712      +/-   ##
============================================
- Coverage     15.17%   15.16%   -0.01%     
- Complexity    11332    11338       +6     
============================================
  Files          5415     5414       -1     
  Lines        474893   475157     +264     
  Branches      57920    57973      +53     
============================================
+ Hits          72046    72076      +30     
- Misses       394792   395022     +230     
- Partials       8055     8059       +4
Flag Coverage Δ
uitests 4.28% <ø> (+<0.01%) ⬆️
unittests 15.89% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@DaanHoogland
Copy link
Contributor

@winterhazel , can you have a look at #10687 (comment) ?

This backport is fine, but I have my doubt on the overall event model for networks. In short, making the usage server robust is fine, but we also need to fix the event flow.

@winterhazel winterhazel mentioned this pull request Apr 15, 2025
Closed
@winterhazel winterhazel marked this pull request as draft April 21, 2025 17:05
@winterhazel winterhazel mentioned this pull request Apr 29, 2025
Open
@DaanHoogland DaanHoogland linked an issue Apr 30, 2025 that may be closed by this pull request
Deletion of LB Rules not reported in Cloud Usage #10789
Open
@DaanHoogland
Copy link
Contributor

@winterhazel do we need more work on this? I think we discussed extra work on other issues/PRs

@DaanHoogland
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13234

@DaanHoogland
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@winterhazel winterhazel marked this pull request as ready for review May 2, 2025 01:47
@winterhazel winterhazel requested a review from rajujith May 2, 2025 01:47
@winterhazel
Copy link
Member Author

@winterhazel do we need more work on this? I think we discussed extra work on other issues/PRs

@DaanHoogland this should be ready now. I made some adjustments here based on the previous discussions:

  • when processing a duplicated NETWORK.CREATE, ignore it. Previously, we were marking the existing helper entry as removed and creating a new one
  • when processing a NETWORK,UPDATE, mark the previous entry as removed and create a new one with the updated information. Previously, the existing entry would have its information overridden, which would result in incorrect usage records (the usage records generated for the period before the network was updated would contain the updated information instead of the real information for that period)

@winterhazel
Copy link
Member Author

@blueorangutan package

@blueorangutan
Copy link

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13235

@blueorangutan
Copy link

[SF] Trillian test result (tid-13192)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 48497 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10712-t13192-kvm-ol8.zip
Smoke tests completed. 132 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_01_redundant_vpc_site2site_vpn Failure 426.09 test_vpc_vpn.py
test_01_vpc_site2site_vpn Failure 298.38 test_vpc_vpn.py

DaanHoogland
DaanHoogland approved these changes May 2, 2025
View reviewed changes
Copy link
Contributor

@DaanHoogland DaanHoogland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some style comments but code looks good!

@DaanHoogland
Copy link
Contributor

@rajujith , can you have a look at this, please?

@winterhazel
Copy link
Member Author

@blueorangutan package

@blueorangutan
Copy link

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13252

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DaanHoogland DaanHoogland DaanHoogland approved these changes

@rajujith rajujith Awaiting requested review from rajujith

Assignees

@rajujith rajujith

Labels
component:usage-server
Projects
None yet
Milestone
4.19.3
Development

Successfully merging this pull request may close these issues.

Deletion of LB Rules not reported in Cloud Usage
4 participants
@winterhazel @DaanHoogland @blueorangutan @rajujith