Create and test new attributes for the NBO parser #1251

weronikazak · 2023-08-10T10:31:16Z

This pull request aims to introduce new attributes for the NBO parser, along with corresponding tests for these attributes.

berquist · 2023-08-11T14:56:22Z

test/testdata

@@ -309,6 +309,7 @@ SP        Psi4        PsiHFSPTest           basicPsi4-1.7               dvb_sp_r
 SP        Psi4        PsiSPTest             basicPsi4-1.7               dvb_sp_rks.out
 SP        QChem       GenericSPTest         basicQChem5.1               dvb_sp.out
 SP        QChem       GenericSPTest         basicQChem5.4               dvb_sp.out
+SP        NBO         NBOSPTest             basicNBO7.0                 basicORCA5.0/dvb_sp.out


This might need to be dvb_sp.nbo.out for just the NBO section.

shivupa · 2023-08-16T19:11:01Z

cclib/parser/data.py

@@ -160,6 +161,7 @@ class ccData:
       "optdone":          Attribute(list,             'done',                        'optimization'),
       "optstatus":        Attribute(numpy.ndarray,    'status',                      'optimization'),
       "polarizabilities": Attribute(list,             'polarizabilities',            'N/A'),
+       "populations":      Attribute(list,             'natural population analysis', 'properties'),


'natural population analysis' -> 'population analysis' since this can hold other types of population analyses.
also I think you forgot to change the type from list to dict here

berquist · 2023-08-17T14:51:00Z

test/testdata

@@ -309,6 +309,7 @@ SP        Psi4        PsiHFSPTest           basicPsi4-1.7               dvb_sp_r
 SP        Psi4        PsiSPTest             basicPsi4-1.7               dvb_sp_rks.out
 SP        QChem       GenericSPTest         basicQChem5.1               dvb_sp.out
 SP        QChem       GenericSPTest         basicQChem5.4               dvb_sp.out
+SP        NBO         NBOSPTest             basicNBO7.0/basicORCA5.0    dvb_sp.nbo.out


Can you put this in alphabetical order regarding the parser name?

berquist · 2023-08-17T14:51:59Z

test/data/testSP.py

+        """Is the number of atoms equal to 3?"""
+        assert self.data.natom == 3
+
+    def testatomcharges(self):


Since only the key name is different from Mulliken, Lowdin, ... in the superclass, can you factor this out into a helper function?

berquist · 2023-08-17T15:02:43Z

test/data/testSP.py

+    """Customized restricted single point unittest"""
+    def testpopulations(self):
+        assert self.data.populations.keys == list(['nao', 'atom', 'no', 'lang', 'type', 'occupancy', 'energy']) 
+        assert isinstance(self.data.populations['nao'], List[int])


I'm not sure this test is getting run, because

>>> l = [1, 2, 3] >>> import typing >>> isinstance(l, typing.List) True >>> isinstance(l, typing.List[int]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/ejberqu/.pyenv/versions/3.10.11/lib/python3.10/typing.py", line 994, in __instancecheck__ return self.__subclasscheck__(type(obj)) File "/Users/ejberqu/.pyenv/versions/3.10.11/lib/python3.10/typing.py", line 997, in __subclasscheck__ raise TypeError("Subscripted generics cannot be used with" TypeError: Subscripted generics cannot be used with class and instance checks

typing is also not for runtime type checking, if you wanted to do that over abstract container types it would be from collections.abc, and then checking element types would be a list comprehension over the container.

berquist · 2023-08-17T15:03:28Z

cclib/parser/nboparser.py

@@ -159,6 +165,9 @@ def extract(self, inputfile, line):

            self.atomcharges["nbo"] = charges

+            if not hasattr(self, "natom"):
+                self.set_attribute('natom', len(self.atomcharges["nbo"]))


berquist · 2023-08-17T15:04:32Z

cclib/parser/nboparser.py

+            }
+
+            if not hasattr(self, "populations"):
+                self.set_attribute('populations', npa_dict)


Didn't we want self.populations["npa"] = {"nao": ..., ...}?

cclib/parser/data.py

berquist · 2023-08-18T14:51:16Z

Something is still happening with the NBO test not being run: https://github.com/cclib/cclib/actions/runs/5903735648/job/16014303794?pr=1251#step:8:873

weronikazak · 2023-08-18T15:22:50Z

Something is still happening with the NBO test not being run: https://github.com/cclib/cclib/actions/runs/5903735648/job/16014303794?pr=1251#step:8:873

Whenever I run the tests, there's a couple of errors popping up

But they appear even after I delete the SP NBO ... record from testdata. I'm running the latest container. Not sure if that's related?

berquist · 2023-08-20T18:44:24Z

Sorry I missed this. I don't develop using the container, though I should, but I'd expect CI to catch this,, since CI is always using the latest image. I'll try reproducing this today.

berquist · 2023-08-20T19:20:16Z

@weronikazak How are you running the tests? I know we need documentation for this.

berquist · 2023-08-20T19:37:58Z

I can reproduce this with

cd /home/eric/development/forks/python/cclib_pr_review
sudo devcontainer up --workspace-folder .
sudo docker exec -it bold_heisenberg /bin/bash  # using the randomized container name
cd /workspaces/cclib_pr_review
python -m pytest test -k 'not test_method'

because .github/scripts/run_pytest.bash doesn't work anymore. (I don't recall having removed --terse.)

echo $? shows 0, which I think is correct. This is happening in regular CI too, where there's no colored printing (https://github.com/cclib/cclib/actions/runs/5798299575/job/15715727150#step:8:581). It likely comes from #1222. It's definitely misleading and needs to be fixed.

I will see what it takes to get the NBO test to run. It probably has something to do with the nested directory in the testdata entry.

berquist · 2023-08-20T19:51:55Z

The stuff to import NBO for testing was missing:

diff --git a/cclib/parser/__init__.py b/cclib/parser/__init__.py
index ab680909..e9e38ac6 100644
--- a/cclib/parser/__init__.py
+++ b/cclib/parser/__init__.py
@@ -25,6 +25,7 @@ from cclib.parser.jaguarparser import Jaguar
 from cclib.parser.molcasparser import Molcas
 from cclib.parser.molproparser import Molpro
 from cclib.parser.mopacparser import MOPAC
+from cclib.parser.nboparser import NBO
 from cclib.parser.nwchemparser import NWChem
 from cclib.parser.orcaparser import ORCA
 from cclib.parser.psi3parser import Psi3
diff --git a/test/data/testSP.py b/test/data/testSP.py
index f198ef2c..d42f4566 100644
--- a/test/data/testSP.py
+++ b/test/data/testSP.py
@@ -604,10 +604,6 @@ class NBOSPTest(GenericSPTest):
     def testnatom(self):
         """Is the number of atoms equal to 3?"""
         assert self.data.natom == 3
-
-    def testatomchargesnbo(self):
-        """Are atomic charges consistent with natom?"""
-        self.testatomcharges()

 class TurbomoleSPTest(GenericSPTest):
     """Customized restricted single point KS unittest"""
diff --git a/test/test_data.py b/test/test_data.py
index 3164c31e..a7c0acf5 100644
--- a/test/test_data.py
+++ b/test/test_data.py
@@ -27,7 +27,7 @@ sys.path.insert(1, os.path.join(__filedir__, 'data'))

 parser_names = [
     "ADF", "DALTON", "FChk", "GAMESS", "GAMESSDAT", "GAMESSUK", "Gaussian", "Jaguar",
-     "Molpro", "Molcas", "MOPAC", "NWChem", "ORCA", "Psi4", "QChem",
+     "Molpro", "Molcas", "MOPAC", "NBO", "NWChem", "ORCA", "Psi4", "QChem",
     "Turbomole",
 ]
 all_parsers = {name: getattr(cclib.parser, name) for name in parser_names}

I think you should replace testatomchargesnbo with a test that ensures "nbo" is a key in atomcharges.

weronikazak · 2023-08-21T10:51:59Z

@weronikazak How are you running the tests? I know we need documentation for this.

I use python -m pytest -v -s --terse test -k "not test_method". Other than the errors above, it seems to be working fine for me

berquist · 2023-08-27T20:02:37Z

I will fix the line ending problem so you don't have to.

berquist · 2023-08-27T20:26:20Z

The merge commit didn't contain any proper changes, so this branch is up-to-date with master.

For the tests, I'm getting

test/test_data.py:116: in getdatafile
    data = logfile.parse()
cclib/parser/logfileparser.py:216: in parse
    data = self.datatype(attributes=self.__dict__)
cclib/parser/data.py:221: in __init__
    self.setattributes(attributes)
cclib/parser/data.py:318: in setattributes
    self.arrayify()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <cclib.parser.data.ccData object at 0x7f34935d9d50>

    def arrayify(self) -> None:
        """Converts appropriate attributes to arrays or lists/dicts of arrays."""

        attrlist = [k for k in self._attrlist if hasattr(self, k)]
        for k in attrlist:
            v = self._attributes[k].type
            precision = 'd'
            if k in self._intarrays:
                precision = 'i'
            if v == numpy.ndarray:
                setattr(self, k, numpy.array(getattr(self, k), precision))
            elif v == list and k in self._listsofarrays:
                setattr(self, k, [numpy.array(x, precision) for x in getattr(self, k)])
            elif v == dict and k in self._dictsofarrays:
                items = getattr(self, k).items()
                pairs = [(key, numpy.array(val, precision)) for key, val in items]
                setattr(self, k, dict(pairs))
>           elif v == dict and k in self._dictsofdicts:
E           AttributeError: 'ccData' object has no attribute '_dictsofdicts'. Did you mean: '_dictsofarrays'?

weronikazak · 2023-08-28T19:10:26Z

I will fix the line ending problem so you don't have to.

Sorry, I just saw this comment!
I've tried to fix it in the meantime, and locally it seems to be working okay. Let's see if it passes the tests

weronikazak · 2023-08-28T19:19:01Z

Hmm, it doesn't work 😕
Do you think it's rebasing issues?

berquist · 2023-08-28T19:49:30Z

Yes. The problem is that you've combined Git operations (checkout, add, commit, or some combination, I'm not sure) on Windows with some inside the container. (The following explanation isn't great and might not be fully correct without reproducing it myself but should be enough.)

When you clone a repo on an operating system, Git by default will transparently not mess up line endings in files if they're different on the remote (GitHub) from your local copy. The settings for how this works are different on Windows than on Linux and macOS. If you have the same checkout that crosses operating systems (like the container bind mount), you can violate these settings. The result is that you can inadvertently change the line endings of every file in the repo.

The best explanation I've found for how to understand this and fix it for Git is https://kuantingchen04.github.io/line-endings/. I have core.autocrlf=input in ~/.gitconfig since I use Linux and sometimes macOS. It sounds like you would want true instead of input if you plan on doing all Git operations Windows-side, though when everything is done in the container (VSC Git extension or terminal) I wouldn't have expected this. If you rebase everything from entirely within the container, the problem might go away?

berquist · 2023-08-28T20:04:14Z

I can see how this works on Windows but it will be a few more hours.

berquist · 2023-08-29T04:59:33Z

I checked out a copy of the repository using the official Git for Windows into my Windows (not WSL) home directory. I don't have a ~/.gitconfig for this. If I open this directory in VSC, it shows no changes; the default is to check out as native line endings (CRLF). Modifying lines As soon as I start the Dev Container,

because Git for Windows checked everything out as CRLF but most files in the repo are entirely LF line endings, and this is running Git from inside the container.

If I set git config --global core.autocrlf=true and then redo everything, the problem remains because the local checkout is still CRLF.

The solution to stop this from happening again is to only clone using WSL so that line endings are consistent with the Dev Container (both are Linux).

The immediate fix is to do the clone in WSL and try rebasing your branch against master from within WSL.

Co-authored-by: Eric Berquist <[email protected]>

berquist

I got rid of the merge commit and the last commit which didn't do what was expected with the line endings. Also the second ~half of commits looked to be the reverse of the first half, so dropped those in the interactive rebase.

shivupa

We can remove rtd fully now that github actions builds docs. ill open a separate issue for that

weronikazak · 2023-08-30T17:57:26Z

I got rid of the merge commit and the last commit which didn't do what was expected with the line endings. Also the second ~half of commits looked to be the reverse of the first half, so dropped those in the interactive rebase.

Thank you!

berquist added this to the v1.8.1 milestone Aug 10, 2023

berquist added the NBO label Aug 10, 2023

berquist reviewed Aug 11, 2023

View reviewed changes

berquist marked this pull request as ready for review August 16, 2023 00:06

shivupa reviewed Aug 16, 2023

View reviewed changes

berquist requested changes Aug 17, 2023

View reviewed changes

berquist mentioned this pull request Aug 20, 2023

Developer guidelines #555

Open

berquist mentioned this pull request Aug 20, 2023

Fix printing false negatives during tests #1255

Open

weronikazak force-pushed the NBO-parser branch from 46b1cf8 to 8efe207 Compare August 27, 2023 06:13

berquist force-pushed the NBO-parser branch from 155a771 to 8efe207 Compare August 27, 2023 20:24

berquist linked an issue Aug 28, 2023 that may be closed by this pull request

New parser request: NBO #1012

Closed

weronikazak force-pushed the NBO-parser branch 2 times, most recently from 0e75bb0 to 514f936 Compare August 28, 2023 18:36

weronikazak added 3 commits August 29, 2023 11:23

Create a new attribute npa

477f314

Change name and type for npa

8d48808

Add tests

6d90c48

weronikazak and others added 10 commits August 29, 2023 11:23

Add tests for NBO

5f76a70

Test atomcharges

419a78b

NBO parser - parse natoms

f33c74e

Update cclib/parser/data.py

86ef524

Co-authored-by: Eric Berquist <[email protected]>

Fix NBO tests

93c8f09

Fix testdata

4c00c8b

Fix attr popultions to be a dictionary

262bf1e

Add references to the NBO parser

9757afa

Fix NBO parsing

f90fb5b

Skip tests

6a3ff8e

weronikazak force-pushed the NBO-parser branch from 8527027 to d095240 Compare August 29, 2023 11:28

Fix dictsofdicts issues

4aeae2f

berquist force-pushed the NBO-parser branch from 04803a7 to 4aeae2f Compare August 29, 2023 20:19

berquist added 2 commits August 29, 2023 16:34

Add NBO to regression.py

d07b261

Read the Docs: remove python.system_packages

9f7dea1

berquist approved these changes Aug 29, 2023

View reviewed changes

berquist requested a review from shivupa August 29, 2023 21:05

shivupa approved these changes Aug 29, 2023

View reviewed changes

shivupa mentioned this pull request Aug 29, 2023

RTD can be removed for github actions #1267

Closed

shivupa merged commit 99c8af6 into cclib:master Aug 29, 2023

berquist mentioned this pull request Mar 11, 2024

Add and update developer guidelines #1355

Merged

23 tasks

Create and test new attributes for the NBO parser #1251

Create and test new attributes for the NBO parser #1251

Uh oh!

Conversation

weronikazak commented Aug 10, 2023

Uh oh!

berquist Aug 11, 2023

Choose a reason for hiding this comment

Uh oh!

shivupa Aug 16, 2023

Choose a reason for hiding this comment

Uh oh!

berquist Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

berquist Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

berquist Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

berquist Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

berquist Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

berquist commented Aug 18, 2023

Uh oh!

weronikazak commented Aug 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

berquist commented Aug 20, 2023

Uh oh!

berquist commented Aug 20, 2023

Uh oh!

berquist commented Aug 20, 2023

Uh oh!

berquist commented Aug 20, 2023

Uh oh!

weronikazak commented Aug 21, 2023

Uh oh!

berquist commented Aug 27, 2023

Uh oh!

berquist commented Aug 27, 2023

Uh oh!

weronikazak commented Aug 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weronikazak commented Aug 28, 2023

Uh oh!

berquist commented Aug 28, 2023

Uh oh!

berquist commented Aug 28, 2023

Uh oh!

berquist commented Aug 29, 2023

Uh oh!

berquist left a comment

Choose a reason for hiding this comment

Uh oh!

shivupa left a comment

Choose a reason for hiding this comment

Uh oh!

weronikazak commented Aug 30, 2023

Uh oh!

Uh oh!

weronikazak commented Aug 18, 2023 •

edited

Loading

weronikazak commented Aug 28, 2023 •

edited

Loading