test syntax please

test syntax please

test syntax please

Here we can monitor the replay:
https://monit-grafana.cern.ch/d/t_jr45h7k/cms-tier0-replayid-monitoring?orgId=11&var-Bin=5m&var-ReplayID=221110152242&var-JobType=All&var-WorkflowType=All

We have 5 paused jobs, al of them with PD HIEmptyBX. The error message is:

cmsRun1
Fatal Exception (Exit Code: 8001)
An exception of category 'NoRecord' occurred while
   [0] Processing  Event run: 361697 lumi: 14 event: 22054465 stream: 2
   [1] Running path 'write_MINIAOD_step'
   [2] Prefetching for module PoolOutputModule/'write_MINIAOD'
   [3] Calling method for module HiEvtPlaneFlatProducer/'hiEvtPlaneFlat'
Exception Message:
No "HeavyIonRPRcd" record found in the EventSetup.

 Please add an ESSource or ESProducer that delivers such a record.

This looks like a GT error to me, is that right @francescobrivio ?

You can find the tarball for one of the jobs here:

/eos/cms/store/logs/prod/recent/PromptReco/PromptReco_Run361697_HIEmptyBX_Tier0_REPLAY_2022_ID221110152242_v10152242/Reco/vocms001.cern.ch-413-3-log.tar.gz

Hi @germanfgv , thanks for the tarball!

I have created a new GT with the missing record: 124X_dataRun3_Prompt_HIreplay2022_v2.

With this new GT, I could run on the tarball you provided without crashes (for as many events as I could :-) ), while I could reproduce the crash right away with the previous GT.

Would you please try this new GT and let us know of problems?

Thanks!

Thanks @malbouis. I'll rerun these jobs with the new GT, and I'll also add another run to be processed from scratch with it.

I'll let you know if we find any problems

I resumed the jobs with the new GTs and they were successful. I added run 361696 to the replay to process from scratch and it has been successful so far.

I resumed the jobs with the new GTs and they were successful. I added run 361696 to the replay to process from scratch and it has been successful so far.

Very good! Thanks German for confirming!
So we (AlCa) will have to update the production Prompt GT to include this fix from Helena!

The replay finished without issues. We processed runs 361696, 361696, and 361779.

test syntax please

At the request of DQM (@rvenditti) I have added a few more dqm_sequences to the HITestRaw[0-23] datasets so the DPGs can check if monitoring their subsystems is possible even in HI conditions and prepare for the actual HI data-taking next year.

Changes:

dqm_sequences=["@commonSiStripZeroBias"],

-->

dqm_sequences=["@commonSiStripZeroBias", "@ecal", "@L1TMon", "@hcal", "@muon", "@jetmet"],

@germanfgv could you please start again the replay with this configuration (I think 1 run is enough) so we can check nothing is crashing with the new dqm_sequences added.

Hi @germanfgv did you have the possibility to launch the reply with updated DQM configuration?

@rvenditti yes. I launched it yesterday evening. It should have finished by now, but I see there are still several idling jobs. I'm going to check what the problem is.

The jobs were idle because, they were issued with low priority. Now they are running and should finish in the next few hours

The replay finished succesfully. I'll add the DQM sequences to #4780