#9888 fixed this and some other issues on 4.20.1. I'll backport the changes to 4.19.3, and work on a normalization script.

@winterhazel thanks for sharing, any timeline on when you can raise the backport PR. Thanks.

@Pearl1594 today 🙂

Regarding the normalization script, though, I hope to look into it within this week.

Thanks @winterhazel! ..

@winterhazel , I noticed a small issue in your solution that I want to discuss. Not sure if this is the place but let me start here at least.

What happens in your solution is that for all CREATE events a remove is forced if a DESTROY event is found for the same network. I think the issue is actually that CREATE events are created on implementation as well, which should be a separate type of event. I am not sure how harmfull this is but we are now hiding the bug instead of solving it.

If you look at for instance VMs, these can have a sequence of events like,
CREATE, START, STOP, START, STOP, DESTROY.
Networks now would have
CREATE, CREATE, CREATE, DESTROY for the same type of actions.
One would expect
CREATE, IMPLEMENT, STOP, IMPLEMENT, STOP, DESTROY

All that said, I am not against backporting your fix. I think in the network case, usage might not require the specifics of implement, though organisations with strict resource billing would.

future work?

Hey @DaanHoogland, thanks for bringing this up.

We're dealing with two different issues here.

The first issue is regarding existing environments. These environments already have multiple CREATE events for the same network, and are seeing (or will see after the next Usage job if we do not address this) duplicated usage record generation. This behavior impacts the usage rating through Quota (because it uses the generated usage records) and through external tools based on the generated usage records.

I don't usually see organizations performing resource usage rating by directly consuming the events from usage_event, only the two options I mentioned. That's why I was concerned in addressing this issue first (through #10712 and a normalization script). Though, now that you pointed it out, I do recognize that this is probably an existing scenario.

Also, I think you may have misunderstood something in my solution by your description, so just making sure that we are aligned on this: my changes mark existing network usage helper entries for a network as removed whenever a CREATE event is processed, independently of there being a DESTROY or not, so that two consecutive CREATE events for the same network do not result in duplicated usage records being generated.

The second issue is what you pointed out: network state changes are not tracked properly. We only have three usage events published for networks, and one of them is published for two different situations:

• NETWORK.CREATE: when the network is created, or implemented.
• NETWORK.DELETE: when the network is deleted.
• NETWORK.UPDATE: when updateNetwork is called for the network.

I agree with you that this should be improved. It would be good to have more event types so that we can properly identify when a network transitioned to Implemented, or from Implemented to Allocated. This would be useful for organizations that perform rating based on the usage events (and Quota as well). However, only improving this would not address existing environments that already have duplicate events and duplicate record generation.

Also, I think you may have misunderstood something in my solution by your description, so just making sure that we are aligned on this: my changes mark existing network usage helper entries for a network as removed whenever a CREATE event is processed, independently of there being a DESTROY or not, so that two consecutive CREATE events for the same network do not result in duplicated usage records being generated.

---

@winterhazel , I largely agree with your previous remarks, except that as you describe above there might be an issue; Older CREATE events shouldn’ t be removed but the newer ones should. The network was actually “ created” on the first create. It may have to be “implemented” multiple times after that , which also lead to CREATE events, but those are not the creation. removing older events instead of the newer ones will lead to missed usage data. (and yes I either misunderstood slightly before , or I do now it seems, but it doesn’t make the overall situation much different)

Make sense?

Older CREATE events shouldn’ t be removed but the newer ones should. The network was actually “ created” on the first create. It may have to be “implemented” multiple times after that , which also lead to CREATE events, but those are not the creation. removing older events instead of the newer ones will lead to missed usage data

@DaanHoogland my changes are not removing CREATE events. Modifying existing records in usage_event is not an option because it will lead to more problems.

I'll try to explain things from the beginning, please tell me if you still see an issue in this:

The way Usage works is around three things: events (from the usage_event table), helper entries (different tables for different resources usage_networks), and the parsers.

• Events are used to create helper entries
• Helper entries indicate a resource's state at a period in time, and are used to generate usage records
• Parsers create usage records (at cloud_usage) based on the helper entries

Let's use the volume event model as an example.

a. Suppose a period between dates A (last time Usage executed) and D (next execution):

1. The user creates a volume at B (between A and D). A CREATE event is saved to usage_event
2. The user, then, resizes the volume at a date C (between B and D). A RESIZE event is saved to usage_event

b. Then, the Usage job runs at D:

1. First, Usage will look through all events created since the last time the job executed and process them
   1.1. It will see that a CREATE event for a volume was issued, so it will create a helper entry in usage_volume to tell the parser that it should generate usage records for the period between the entry's created field (B) until the deleted field (currently null, that is, until the current moment D)
   1.2. It will see that a RESIZE event for a volume was issued. To identify that there was a change in the volume, it creates another entry in usage_volume with created set to C, and marks the previous one as removed at date C

2. After this, the "parsing" process will execute:
   2.1. The volume parser will check usage_volume for entries that apply to the period between A and D
   2.1.1. It will see that there is one entry that starts at B and ends at C, so it will generate an usage record that contains how much time there is between B and C, and information about the volume (before the resize) that are in the helper entry
   2.1.2. It will see that there is one entry that starts at C and does not have an end date, so it will generate an usage record that contains how much time there is between C and D, and information about the volume (after the resize) that are in the helper entry

My changes implement b.1.2 for the network event processing. That is, Usage will mark previous helper entries (not events!) as removed when it sees a CREATE event for network, but there are already existing helper entries.

This does not lead to missing usage data. The usage records will be generated as intended, as exemplified in this comment, and events remain unchanged.

@winterhazel , I understand all that and I am just talking about the usage_networks intermediate table because NETWORK.CREATE-events are created for different events, some of which are not the actual creation (in symmetry with NETWORK.DELETE), we are now marking all but the last one as removed. as you said

"Usage will mark previous helper entries (not events!) as removed when it sees a CREATE event for network, but there are already existing helper entries.”

My point is that it should not. It should consider the oldest one as legitimate (for networks!) because of the bug we have that after creation of a network more NETWORK.CREATE-event-records are created. Only the oldest one is legitimate.

I do not know what the consequences for usage are, but we should address this creation of events (at least) as well.

@DaanHoogland ah, I think I now understood what you're proposing. It seems good for me.

I'll change the current code on NETWORK.CREATE event processing to not create new helper entries for a network when an active one already exists, and make the normalization keep the first created entry as not removed instead of the last one.

I'll also take a look at the duplicated NETWORK.CREATE event creation.

thanks @winterhazel , I will have a look as well. I think adding "NETWORK.ALLOCATION" and "NETWORK.DEALLOCATION" would be good (intuition) but not sure if that makes sense.

@DaanHoogland I looked into the network Usage event model. Turns out that I was somewhat wrong in my previous comment regarding when NETWORK.CREATE and NETWORK.UPDATE are published:

• NETWORK.CREATE is not published when the network is created. Instead, it is published only after the network is implemented, which does not match what Usage expects
• NETWORK.UPDATE is also published to indicate that the network transitioned to Implemented; however, in a situation that does not make much sense:
  • when the method that implements the network is called after the network has already been implemented, such as when deploying a second VM on the network

This means that:

• networks that were created, but not implemented, do not generate usage records
• implemented networks can generate usage records indicating that their state is Allocated in some situations, such as when only a single VM has been associated with the network

I opened #10755 to fix these, and to track when the network goes from Implemented to Allocated, but have not properly tested it yet (will do and describe the tests tomorrow).

I think adding "NETWORK.ALLOCATION" and "NETWORK.DEALLOCATION" would be good (intuition) but not sure if that makes sense.

@DaanHoogland I think we can keep using NETWORK.UPDATE to indicate state change for networks, as is done currently. We just need to fix when the events are published.

• NETWORK.CREATE is not published when the network is created. Instead, it is published only after the network is implemented, which does not match what Usage expects

This is not what I observed. I’ll have another look. I saw the NETWORK.CREATE being published on both create and implement. @rajujith, what did you see ?

correct @DaanHoogland, NETWORK.CREATE on create and implement. NETWORK.CREATE event also logged when the network is re-implemented when first VM is created after GC (network.gc.interval config is set) - multiple records are created when the network is GC-ed and re-implemented several times.

@DaanHoogland @sureshanaparti just making sure:

• are you checking the table cloud.usage_event, and not cloud.event? A NETWORK.CREATE is inserted on cloud.event when the network is created (allocated), but not on cloud.usage_event
• are you checking the table before the network is ever implemented? (while its state is Allocated)

I tested this multiple times on both the 4.19.2.0 and 4.20.0.0 releases, but nothing is inserted on cloud.usage_event after creating the network.

Also, the only place I found NETWORK.CREATE being inserted on cloud.usage_event are these two lines, which are after the network is implemented:

If you could confirm these and provide me where the event is being inserted on cloud.usage_event (if that's the case) it would be great.

• NETWORK.CREATE is not published when the network is created. Instead, it is published only after the network is implemented, which does not match what Usage expects

This is not what I observed. I’ll have another look. I saw the NETWORK.CREATE being published on both create and implement. @rajujith, what did you see ?

I do not see any usage event when I create an isolated network, i.e when it isin the 'Allocated; state. I see the NETWORK.CREATE when the network is implemented. @winterhazel mentioned above the current behavior.

For isolated networks and VPC tiers, a usage event NETWORK.CREATE is created in the cloud.usage_event table whenever the network transitions from 'Allocated' to 'Implemented' state. A NETWORK.DELETE event is created when the network is deleted.

ok, thanks @rajujith , I thought I had seen it one time extra, but won’t bother to reproduce.

@DaanHoogland @sureshanaparti just making sure:

• are you checking the table cloud.usage_event, and not cloud.event? A NETWORK.CREATE is inserted on cloud.event when the network is created (allocated), but not on cloud.usage_event

correct, NETWORK.CREATE is added on cloud.event table, and not on cloud.usage_event.

• are you checking the table before the network is ever implemented? (while its state is Allocated)

correct

I tested this multiple times on both the 4.19.2.0 and 4.20.0.0 releases, but nothing is inserted on cloud.usage_event after creating the network.

yes, I observed the same

@winterhazel comments inline ^^^

My observations below:

Created Isolated Network (Network in Allocated state) =>

mysql> SELECT * FROM cloud.event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| id   | uuid                                 | type           | state     | description                                              | user_id | account_id | domain_id | resource_id | resource_type | created             | level | start_id | parameters | archived | display |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| 1129 | 4af06f76-9788-40a9-80a4-f10da4dafb0f | NETWORK.CREATE | Completed | Successfully completed creating network. Network Id: 206 |       2 |          2 |         1 |         206 | Network       | 2025-04-24 09:14:41 | INFO  |        0 | NULL       |        0 |       1 |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
1 row in set (0.00 sec)

mysql> SELECT * FROM cloud.usage_event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
Empty set (0.00 sec)

Deployed VM in Isolated Network created above (Network in Implemented state) =>

mysql> SELECT * FROM cloud.event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| id   | uuid                                 | type           | state     | description                                              | user_id | account_id | domain_id | resource_id | resource_type | created             | level | start_id | parameters | archived | display |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| 1129 | 4af06f76-9788-40a9-80a4-f10da4dafb0f | NETWORK.CREATE | Completed | Successfully completed creating network. Network Id: 206 |       2 |          2 |         1 |         206 | Network       | 2025-04-24 09:14:41 | INFO  |        0 | NULL       |        0 |       1 |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
1 row in set (0.00 sec)

mysql> SELECT * FROM cloud.usage_event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id  | type           | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| 365 | NETWORK.CREATE |          2 | 2025-04-24 09:22:51 |       1 |         206 | testnw02      |          10 |        NULL | NULL | NULL          |         0 |         NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
1 row in set (0.00 sec)

Stopped the VM deployed above (Network back to Allocated state, after NetworkGarbageCollector triggered) =>

mysql> SELECT * FROM cloud.event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| id   | uuid                                 | type           | state     | description                                              | user_id | account_id | domain_id | resource_id | resource_type | created             | level | start_id | parameters | archived | display |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| 1129 | 4af06f76-9788-40a9-80a4-f10da4dafb0f | NETWORK.CREATE | Completed | Successfully completed creating network. Network Id: 206 |       2 |          2 |         1 |         206 | Network       | 2025-04-24 09:14:41 | INFO  |        0 | NULL       |        0 |       1 |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
1 row in set (0.00 sec)

mysql> SELECT * FROM cloud.usage_event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id  | type           | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| 365 | NETWORK.CREATE |          2 | 2025-04-24 09:22:51 |       1 |         206 | testnw02      |          10 |        NULL | NULL | NULL          |         0 |         NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
1 row in set (0.00 sec)

Started the VM (Network in Implemented state) =>

mysql> SELECT * FROM cloud.event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| id   | uuid                                 | type           | state     | description                                              | user_id | account_id | domain_id | resource_id | resource_type | created             | level | start_id | parameters | archived | display |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| 1129 | 4af06f76-9788-40a9-80a4-f10da4dafb0f | NETWORK.CREATE | Completed | Successfully completed creating network. Network Id: 206 |       2 |          2 |         1 |         206 | Network       | 2025-04-24 09:14:41 | INFO  |        0 | NULL       |        0 |       1 |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
1 row in set (0.00 sec)

mysql> SELECT * FROM cloud.usage_event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id  | type           | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| 365 | NETWORK.CREATE |          2 | 2025-04-24 09:22:51 |       1 |         206 | testnw02      |          10 |        NULL | NULL | NULL          |         0 |         NULL |
| 370 | NETWORK.CREATE |          2 | 2025-04-24 09:33:05 |       1 |         206 | testnw02      |          10 |        NULL | NULL | NULL          |         0 |         NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
2 rows in set (0.00 sec)

nice! @winterhazel , I think we can mark your PR #10755 as fixing this issue. Than ks a lot.

fixed in #10755