Add sites configuration to easily switch storage sites #4596
Conversation
|
Container Tests |
| streamerPNN = "T0_CH_CERN_Disk" | ||
|
|
||
| addSiteConfig(tier0Config, "T0_CH_CERN_Disk", | ||
| siteLocalConfig="/cvmfs/cms.cern.ch/SITECONF/T0_CH_CERN/JobConfig/site-local-config.xml", |
There was a problem hiding this comment.
Due that this siteLocalConfig and overrideCatalog have a default value within the function addSiteConfig, Is it mandatory to use it again here in the OfflineConfig file??
There was a problem hiding this comment.
Yes, it is not mandatory. But I think is worth adding it explicitly, so it is clear to everyone what values we are using. Like when we add a default Repack config or default dataset config.
jhonatanamado
left a comment
There was a problem hiding this comment.
I added some minor comments. The only mandatory thing to change is the value for processingSite. Please add the correct value and update the ProducitonOfflineConfig file :)
|
Container Tests |
|
@drkovalskyi , I tested this PR with a replay, checked the changes and everything looks good. Any comments ? |
|
Looks reasonable to me. Can we fix the checks so that we don't have failed test before we merge? |
|
run unit test please |
|
Container Tests |
|
@drkovalskyi there was a few issues with the unit test set up. I fixed them. Now it is working and makes more sense. We can talk about them tomorrow. For now I think we can merge this PR. |
Before #4566, choosing a particular processing site implied that we were using the same site as storage. Now we have the ability to process data in T2_CH_CERN while storing output in different disk sites. This feature is currently being use in production to write output data to T0_CH_CERN_Disk and to test EOS configurations with EOS_PILOT.
Given this development, our previous way of configuring sites needed to be reworked. This PR does that by adding the following fields to the Tier0 configuration:
Added StorageSite as a Global configuration parameter aside from ProcessingSite: This makes it more clear to pick different storage sites.
Added Site configurations: Now we can specify relevant information when using a new site
Now TFC and site local config files are overriten only when there is a difference between ProcessingSite and StorageSite