Set PNN instead of PSNs for wmbs file locations #8280

amaltaro · 2017-10-27T11:58:34Z

Fix #8258
First commit has some improvements/unused code changes, which I applied while debugging that issue.
Second commit changes all wmbs file setters to set the PNN instead of the location (PSN mapped from PNN).

I'm trying not to change any logic in the code, but only change what we store in the database and adapt getters/setters.

@ticoann and @hufnagel can you please check whether I'm going at the right direction

Code is completely untested btw.

cmsdmwmbot · 2017-10-27T12:17:06Z

Jenkins results:

Unit tests: failed
- 441 new failures
- 1 tests no longer failing
- 13 changes in unstable tests
Pylint check: failed
- 28 warnings and errors that must be fixed
- 6 warnings
- 154 comments to review
Pycodestyle check: succeeded
- 287 comments to review
Python3 compatibility checks: failed
- fails python3 compatibility test
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6294/artifact/artifacts/PullRequestReport.html

amaltaro · 2017-10-27T12:25:41Z

Ho ho ho , just 441 failures...

cmsdmwmbot · 2017-10-27T13:36:28Z

Jenkins results:

Unit tests: failed
- 444 new failures
- 1 tests no longer failing
- 13 changes in unstable tests
Pylint check: failed
- 28 warnings and errors that must be fixed
- 6 warnings
- 154 comments to review
Pycodestyle check: succeeded
- 287 comments to review
Python3 compatibility checks: failed
- fails python3 compatibility test
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6295/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2017-10-27T14:50:59Z

Jenkins results:

Unit tests: failed
- 442 new failures
- 12 changes in unstable tests
Pylint check: failed
- 28 warnings and errors that must be fixed
- 6 warnings
- 154 comments to review
Pycodestyle check: succeeded
- 287 comments to review
Python3 compatibility checks: failed
- fails python3 compatibility test
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6296/artifact/artifacts/PullRequestReport.html

amaltaro · 2017-10-27T14:55:01Z

And it keeps failing for me on the oracle backend as well:

Exception Class: WMException
Message: Problem creating database table 

ALTER TABLE wmbs_file_location ADD
                (CONSTRAINT fk_location_location FOREIGN KEY(location)
                   REFERENCES wmbs_location_pnns(pnn) ON DELETE CASCADE)

(DatabaseError) ORA-02270: no matching unique or primary key for this column-list
 'ALTER TABLE wmbs_file_location ADD\n                (CONSTRAINT fk_location_location FOREIGN KEY(location)\n                   REFERENCES wmbs_location_pnns(pnn) ON DELETE CASCADE)' {}
	ModuleName : WMCore.Database.DBCreator
	MethodName : execute
	ClassInstance : None
	FileName : /data/amaltaro/WMAgent/unittestdeploy/wmagent/1.1.5.pre5/sw/slc6_amd64_gcc493/cms/wmagent/1.1.5.pre5/lib/python2.7/site-packages/WMCore/Database/DBCreator.py
	ClassName : None
	LineNumber : 90
	ErrorNr : WMCORE-2

any clue @hufnagel ?

cmsdmwmbot · 2017-10-27T14:59:18Z

Jenkins results:

Unit tests: failed
- 443 new failures
- 12 changes in unstable tests
Pylint check: failed
- 28 warnings and errors that must be fixed
- 6 warnings
- 154 comments to review
Pycodestyle check: succeeded
- 287 comments to review
Python3 compatibility checks: failed
- fails python3 compatibility test
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6297/artifact/artifacts/PullRequestReport.html

hufnagel · 2017-10-27T22:57:17Z

I have no clue what this schema is supposed to be honestly. Doesn't make any sense to me. Location still is mapped to a fixed set of PNNs, but for the file location you don't even store PNN, you store what ? A string that doesn't map to anything ?

hufnagel · 2017-10-27T22:58:43Z

Remove location from wmbs_file_location, replace it with a pnn (id) and have a separate table with the allowed list of pnn and a corresponding FK id in the wmbs_location table.

hufnagel · 2017-10-27T22:59:40Z

If you still think your schema works, can you give me an example how you would inject data into this ?

amaltaro · 2017-10-30T08:21:18Z

Location still is mapped to a fixed set of PNNs, but for the file location you don't even store PNN, you store what ? A string that doesn't map to anything ?

Yes, file location is mapped to a set of PNNs (the pnn string, not their id or mapped location).
Then this PNN string remains with the same functionality, which is mapping to location(s) (from wmbs_location).

Cutting a long story short, we no longer map pnn to location and set them as file location. We set PNN(s) as file location and map it to their location when needed.

Remove location from wmbs_file_location, replace it with a pnn (id) and have a separate table with the allowed list of pnn and a corresponding FK id in the wmbs_location table.

I'm not sure I follow you. Isn't it exactly what we have at the moment? PNNs with a corresponding FK id in wmbs_location ?

If you still think your schema works, can you give me an example how you would inject data into this ?

Check these DAOs:
https://github.com/dmwm/WMCore/pull/8280/files#diff-be8f9b7fa79ec84aff1c627bebc46142R14
https://github.com/dmwm/WMCore/pull/8280/files#diff-abb83a29831c5f9178faefdbe4e6df0dR14

they were already using the PNN as query binds (but mapping to a location - aka site_name).

amaltaro · 2017-10-30T09:00:15Z

And I think I know why it fails... it seems a foreign key has to reference either a primary key OR a unique key in the parent table. Since wmbs_location_pnns has a many to many mapping, it doesn't work.

I guess the only option is really creating another table with unique id and pnns, then keep wmbs_location_pnns just to map location (id - site_name - from wmbs_location) to pnn (id - pnn - from wmbs_pnns)

hufnagel · 2017-10-30T19:53:25Z

Remove location from wmbs_file_location, replace it with a pnn (id) and have a separate table with the allowed list of pnn and a corresponding FK id in the wmbs_location table.

I'm not sure I follow you. Isn't it exactly what we have at the moment? PNNs with a corresponding FK id in wmbs_location ?

Still won't work. You can't store PNN or even PNN id in wmbs_location since that hardcodes a 1 to many relationship between location(where you submit jobs) and pnn (where you store files). Which is not correct, it's a many to many relationship.

Which is the conclusion you came to, so I guess I can review a new version soon :-) ?

amaltaro · 2017-11-03T15:30:23Z

A pretty bad jenkins output is coming soon, just don't bother about it. I still have to update all DAOs.
@hufnagel you might have a look at the schema changes though (Create and CreateBase modules).

ticoann · 2017-11-03T16:41:53Z

src/python/WMCore/WMBS/CreateWMBSBase.py

            """CREATE TABLE wmbs_file_location (
               fileid   INTEGER NOT NULL,
-               location INTEGER NOT NULL,
+               location VARCHAR(255),


Shouldn't this be integer?

Also, this should be pnn, not location.

ticoann · 2017-11-03T16:43:04Z

src/python/WMCore/WMBS/Oracle/Create.py

            """CREATE TABLE wmbs_file_location (
                 fileid   INTEGER NOT NULL,
-                 location INTEGER NOT NULL
+                 location VARCHAR(255)


shouldn't this be integer?

Also, this should be pnn, not location.

ticoann · 2017-11-03T19:58:18Z

src/python/WMCore/WMBS/CreateWMBSBase.py

               FOREIGN KEY(fileid)   REFERENCES wmbs_file_details(id)
                 ON DELETE CASCADE,
-               FOREIGN KEY(location) REFERENCES wmbs_location(id)
+               FOREIGN KEY(location) REFERENCES wmbs_location_pnns(pnn)


Is there a reason to use foreign on wmbs_location_pnns instead of wmbs_pnns

ticoann · 2017-11-03T20:01:33Z

src/python/WMCore/WMBS/MySQL/Files/GetLocationBulk.py

+    sql = """SELECT wfl.location as pnn, wfl.fileid as id
+               FROM wmbs_file_location wfl
+               WHERE wfl.fileid = :id
+            """


If you need to return the same thing it should be pnn string not pnn integer.
need to join wmbs_pnns table.

ticoann · 2017-11-03T20:03:10Z

src/python/WMCore/WMBS/MySQL/Files/SetLocationForWorkQueue.py

-                       FROM wmbs_location_pnns wls, wmbs_file_details
-                       WHERE wls.pnn = :location
-                       AND wmbs_file_details.lfn = :lfn"""
+                   VALUES ((SELECT id FROM wmbs_file_details WHERE lfn = :lfn ), :location)"""


same thing here need to join wmbs_pnns table to get location id (which is pnn id)

ticoann · 2017-11-03T20:09:40Z

This seems very error prone changes although I agree with having separate pnn tables. There are too many things to changes. I wish there will be better solution. 😄

amaltaro · 2017-11-06T13:35:46Z

@hufnagel Dirk, do you use this DAO in the T0 agent:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMBS/MySQL/Jobs/LoadFileLocations.py
?

which is only used here:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMBS/Job.py#L225

and nothing calls that Job method.

amaltaro · 2017-11-06T13:46:58Z

src/python/WMCore/WMBS/MySQL/Subscriptions/GetAvailableFiles.py


 class GetAvailableFiles(DBFormatter):
-    sql = """SELECT wmbs_sub_files_available.fileid, wls.pnn AS pnn
+    sql = """SELECT wmbs_sub_files_available.fileid, wmbs_file_location.location AS pnn


Not clear to me why GetAvailable returns PNN while GetAcquired returns PSN...

amaltaro · 2017-11-06T13:49:04Z

And a few more DAOs not used anywhere:
Subscriptions/GetAcquiredFiles
Subscriptions/GetAcquiredFilesByLimit
Subscriptions/GetAvailableFilesByRun
Subscriptions/GetAvailableFilesByLimit

amaltaro · 2017-11-06T14:05:38Z

I've found several DAOs not used. I'll create a new issue and a new PR for that removal and we can follow up from there.

cmsdmwmbot · 2017-11-30T20:25:53Z

Jenkins results:

Unit tests: failed
- 1 new failures
- 3 changes in unstable tests
Pylint check: failed
- 48 warnings and errors that must be fixed
- 6 warnings
- 619 comments to review
Pycodestyle check: succeeded
- 1218 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6647/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2017-11-30T21:03:40Z

Jenkins results:

Unit tests: failed
- 1 new failures
- 3 changes in unstable tests
Pylint check: failed
- 48 warnings and errors that must be fixed
- 6 warnings
- 619 comments to review
Pycodestyle check: succeeded
- 1218 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6649/artifact/artifacts/PullRequestReport.html

amaltaro · 2017-11-30T21:07:22Z

Thank you very much for the review, Seangchan and Dirk!
Seangchan, I updated the DAO issue you pointed out and now I see a correct submit thresholds dictionary. In addition to that, DMWM tests ran successfully in a MySQL agent.

I'm going to setup an oracle agent now and, if everything is Ok, I'll remove the label and ask you to merge it tomorrow.

amaltaro · 2017-12-01T05:38:02Z

test this please

amaltaro · 2017-12-01T10:12:49Z

test this please

amaltaro · 2017-12-01T10:16:22Z

Everything is awesome!!! All DMWM templates tested in both oracle and mysql wmagents (also central services running my private tag) and there are no errors in the logs, nothing crashed and workflows succeeded.

@ticoann I think we can merge it once WMCore gets deployed on cmsweb production (but better merging this one first to avoid conflicts :))

cmsdmwmbot · 2017-12-01T10:44:23Z

Jenkins results:

Unit tests: succeeded
- 1 tests no longer failing
- 2 changes in unstable tests
Pylint check: failed
- 48 warnings and errors that must be fixed
- 6 warnings
- 619 comments to review
Pycodestyle check: succeeded
- 1218 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6660/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2017-12-01T22:34:08Z

Jenkins results:

Unit tests: succeeded
- 1 tests no longer failing
- 2 changes in unstable tests
Pylint check: failed
- 47 warnings and errors that must be fixed
- 6 warnings
- 140 comments to review
Pycodestyle check: succeeded
- 187 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6661/artifact/artifacts/PullRequestReport.html

amaltaro · 2017-12-02T08:06:03Z

Not a bad pylint fix, from
1218 comments to review
to
187 comments to review

I'm squashing these last two commits and it's good to go.

Create index for wmbs_location_pnns pnn Proper ordering of the tables creation typo on table name Add unneeded files back; will be deleted in a separate PR Add new wmbs_pnns table and update other location tables Seangchan and Dirk suggestions for the new schema fix syntax errors in schema creation rename oracle fk constraint No need for a sequence for wmbs_location_pnns Update all DAOs and logic for the new wmbs_pnns table Update resource control New DAOs fix a few more DAOs ignore duplicate wmbs_location_pnns inserts More ResourceControl DAO fixes and site/se info fix GetLocationBulk dao select id from sequence for wmbs_pnns Fix oracle Locations.New dao Fixed Oracle.Locations.New DAO for wmbs_location_pnns apply SEangchans suggestion

fix logging test pylint fixes

cmsdmwmbot · 2017-12-02T08:38:23Z

Jenkins results:

Unit tests: failed
- 1 new failures
- 2 changes in unstable tests
Pylint check: failed
- 47 warnings and errors that must be fixed
- 6 warnings
- 140 comments to review
Pycodestyle check: succeeded
- 187 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6662/artifact/artifacts/PullRequestReport.html

ticoann · 2017-12-02T14:17:48Z

test this please

cmsdmwmbot · 2017-12-02T14:45:44Z

Jenkins results:

Unit tests: failed
- 1 new failures
- 4 changes in unstable tests
Pylint check: failed
- 47 warnings and errors that must be fixed
- 6 warnings
- 140 comments to review
Pycodestyle check: succeeded
- 187 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6663/artifact/artifacts/PullRequestReport.html

amaltaro · 2017-12-04T11:13:15Z

test this please

cmsdmwmbot · 2017-12-04T11:41:41Z

Jenkins results:

Unit tests: failed
- 1 new failures
- 29 tests no longer failing
- 3 tests added
- 2 changes in unstable tests
Pylint check: failed
- 47 warnings and errors that must be fixed
- 6 warnings
- 140 comments to review
Pycodestyle check: succeeded
- 187 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6664/artifact/artifacts/PullRequestReport.html

ericvaandering · 2017-12-04T15:27:28Z

test this please

ericvaandering · 2017-12-04T15:27:39Z

Baseline was screwed up this weekend

cmsdmwmbot · 2017-12-04T15:50:02Z

Jenkins results:

Unit tests: succeeded
- 1 tests no longer failing
Pylint check: failed
- 47 warnings and errors that must be fixed
- 6 warnings
- 140 comments to review
Pycodestyle check: succeeded
- 187 comments to review
Python3 compatibility checks: succeeded
- there are suggested fixes for newer python3 idioms

Details at https://cmssdt.cern.ch/jenkins/view/All/job/DMWM-WMCore-PR-test/6666/artifact/artifacts/PullRequestReport.html

amaltaro · 2017-12-04T21:01:19Z

@ticoann we better cut a new tag now, just in case :)

ticoann · 2017-12-04T21:26:31Z

I thought this works perfectly fine. 😄 I will cut a new tag 1.1.10.pre1 and run the test.

amaltaro · 2017-12-04T21:33:09Z

Don't bother running tests Seangchan ;)

amaltaro added the PR: Do not merge yet label Oct 27, 2017

amaltaro self-assigned this Oct 27, 2017

amaltaro force-pushed the file-location-worker branch from b2aa6eb to 1c7c1a3 Compare October 27, 2017 13:21

amaltaro force-pushed the file-location-worker branch 2 times, most recently from b7fd144 to 156f39e Compare October 27, 2017 14:41

ticoann reviewed Nov 3, 2017

View reviewed changes

amaltaro commented Nov 6, 2017

View reviewed changes

amaltaro force-pushed the file-location-worker branch from 38a81ba to ed04683 Compare November 6, 2017 14:26

amaltaro mentioned this pull request Nov 6, 2017

Unused DAOs to be removed #8301

Closed

amaltaro force-pushed the file-location-worker branch from b048b07 to ada15c4 Compare November 30, 2017 20:38

amaltaro removed the PR: Do not merge yet label Dec 1, 2017

amaltaro added 3 commits December 2, 2017 09:06

Just some minor improvements to file location handling

c9cec52

Fix a few last unittests, none related to these changes though

9fe2e40

fix logging test pylint fixes

amaltaro force-pushed the file-location-worker branch from 9115c77 to 9fe2e40 Compare December 2, 2017 08:06

ticoann merged commit 78860fe into dmwm:master Dec 4, 2017

ticoann mentioned this pull request Dec 4, 2017

Create the change for Tier0 on PNN, PSN mapping. dmwm/T0#4401

Closed

amaltaro mentioned this pull request Dec 8, 2017

Set PNN instead of PSNs for wmbs file locations dmwm/T0#4402

Merged

amaltaro deleted the file-location-worker branch April 10, 2018 13:16

Set PNN instead of PSNs for wmbs file locations #8280

Set PNN instead of PSNs for wmbs file locations #8280

Uh oh!

Conversation

amaltaro commented Oct 27, 2017

Uh oh!

cmsdmwmbot commented Oct 27, 2017

Uh oh!

amaltaro commented Oct 27, 2017

Uh oh!

cmsdmwmbot commented Oct 27, 2017

Uh oh!

cmsdmwmbot commented Oct 27, 2017

Uh oh!

amaltaro commented Oct 27, 2017

Uh oh!

cmsdmwmbot commented Oct 27, 2017

Uh oh!

hufnagel commented Oct 27, 2017

Uh oh!

hufnagel commented Oct 27, 2017

Uh oh!

hufnagel commented Oct 27, 2017

Uh oh!

amaltaro commented Oct 30, 2017

Uh oh!

amaltaro commented Oct 30, 2017

Uh oh!

hufnagel commented Oct 30, 2017

Uh oh!

amaltaro commented Nov 3, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ticoann commented Nov 3, 2017

Uh oh!

amaltaro commented Nov 6, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amaltaro commented Nov 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amaltaro commented Nov 6, 2017

Uh oh!

cmsdmwmbot commented Nov 30, 2017

Uh oh!

cmsdmwmbot commented Nov 30, 2017

Uh oh!

amaltaro commented Nov 30, 2017

Uh oh!

amaltaro commented Dec 1, 2017

Uh oh!

amaltaro commented Dec 1, 2017

Uh oh!

amaltaro commented Dec 1, 2017

Uh oh!

cmsdmwmbot commented Dec 1, 2017

Uh oh!

cmsdmwmbot commented Dec 1, 2017

Uh oh!

amaltaro commented Dec 2, 2017

Uh oh!

cmsdmwmbot commented Dec 2, 2017

Uh oh!

ticoann commented Dec 2, 2017

Uh oh!

cmsdmwmbot commented Dec 2, 2017

amaltaro commented Nov 6, 2017 •

edited

Loading