Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

dhomeier
Copy link
Contributor

Fixes #931 using the latest available upstream versions for Python up to 3.10 and < 3.7, respectively.
I have not had a recent system with the older Python versions available to test the 1.18.1 package on.
The point of failure seems to have been adding docutils/utils to the local PYTHONPATH. This overshadowed system modules like math, and I don't see it required – the local utils versions are already imported with the their relative path, e.g. as from docutils.utils import math.

@dhomeier dhomeier added bug new upstream Package has an updated upstream version labels Dec 26, 2022
@dhomeier dhomeier requested a review from dmacks December 26, 2022 22:02
@nieder nieder requested a review from clepple December 27, 2022 09:54
@nieder
Copy link
Member

nieder commented Dec 27, 2022

  1. Why does the newer version change the variant naming format of the binaries from %type_raw[] to %type_pkg[] ? Won't that cause update-alternatives chaos?
  2. Do we really need to keep py37-py310 alive in the older v0.18.1?
  3. According to the README, 0.15.x is the last version supporting Python 2.6, 3.3 and 3.4, so we'll need another .info for py34.

@dhomeier
Copy link
Contributor Author

Think I got things mixed up with the existing naming scheme, which actually ends up as docutils-3.10.py rather than docutils3.10, docutils-3.10, docutils-py310 or anything else typically in use. I find it very confusing, but you're probably right that update-alternatives will struggle handling this.

The 1.18.1 version was an option to have all variants in the same version, but if py34 needs to be kept alive, several versions will be required. Perhaps easiest just to keep docutils-py34-1.14; or keep all the variants < py37 at 1.15 – I don't really care which.

@nieder
Copy link
Member

nieder commented Dec 27, 2022

I'll have a 10.14 machine available next week (currently 12.6). I can work on this then. @clepple do you know any issues with updating docutils-py?

@clepple
Copy link
Contributor

clepple commented Dec 27, 2022

@nieder TBH, I'm still a little ways away from being able to test this myself (had issues reinstalling Fink on a 10.15 box), but I was mainly tracking docutils for Trac and Django, and the latter seems to be better handled by Python virtualenvs (IMHO).

The only fixed dependencies I found by running git grep in this repo (admittedly a brittle check) are lensfun1-shlibs (docutils-py38) and several versions of mpv (-py38 and -py39, notably). So not sure if extra effort needs to be applied for -py34. I'm not familiar enough with Fink deprecation policy to know whether this PR checks those boxes properly.

@dhomeier thanks for putting this PR together!

@dhomeier
Copy link
Contributor Author

@clepple thanks for checking the dependent packages; I also found sphinx-py and some of it's extension modules, which bears down quite some way into pythonmods doc builds.

I managed to get a 17.7 system up after all with python34 and python35 test builds and found

  1. docutils-py34-0.18.1 does build successfully, with a couple of warnings, but none I could not also spot with py35 and py39; generating its own docs also succeeds.
  2. The present update-alternatives setup with configuring a slave for each file turned out even more complex, as this was installing a %p/etc/alternatives/rst2html-py link to %p/bin/rst2html-3.5.py for %p/bin/rst2html.py etc.
  3. I was also worried that renaming them could cause problems on fink update; but installing an update with the new naming scheme over the 0.14 package with the old one just cleared the old names and links without problems.

I am proposing, in the last commit, to switch to install all other scripts as slaves to rst2html, thus simplifying PreRmScript. This would certainly need some more testing, though.

@dmacks
Copy link
Member

dmacks commented Jan 2, 2023

As long as the change of the package-specific bin program naming scheme (57560f9) allows clean upgrade (existing bin/rst2html link automatically updates to the new-named target), I'm not concerned about it. We don't have a guideline about how to name those sorts of variant-specific files that are completely handled by update-alternatives magic. Using --slave certainly seems cleaner, and it's also sensible to keep a suite of tools in sync, so again if it doesn't confuse u-a during upgrades, it's just maintainers'-choice.

@TheSin-
Copy link
Member

TheSin- commented May 31, 2023

At first glance this looks good, since I can't install the old version I can not test upgrade path for it.

Can we get this into master?

@TheSin-
Copy link
Member

TheSin- commented Jun 2, 2023

Anyone?? We really need docutils as it's builddep for lots of packages. Anyone able to test these packages with an upgrade path against the current versions?

@dak180
Copy link
Member

dak180 commented Jun 6, 2023

@TheSin- and @dhomeier :

Installing docutils script to /opt/sw/src/fink.build/root-docutils-py39-0.19-1/opt/sw/bin

for i in *.txt ; do
  htmldoc=${i%txt}html
  echo "Making $htmldoc"
  PYTHONPATH=.:docutils /opt/sw/bin/python3.9 tools/rst2html.py $i $htmldoc
done
Making BUGS.html
Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
AttributeError: module 'io' has no attribute 'open'

Current thread 0x0000000102f50580 (most recent call first):
<no Python frame>
### execution of /tmp/fink.uQFKW failed, exit code 1
### execution of /tmp/fink.eR5tZ failed, exit code 1
Removing runtime build-lock...
Removing build-lock package...
/opt/sw/bin/dpkg-lockwait -r fink-buildlock-docutils-py39-0.19-1
Package manager version: 0.45.99.git
Distribution version: selfupdate-rsync Tue Jun  6 15:19:10 2023, 12.0, arm64
Trees: local/main remote/dak180 stable/main local/injected
Xcode.app: 14.2
Xcode command-line tools: 14.2.0.0.1.1668646533
Max. Fink build jobs:  1

@TheSin-
Copy link
Member

TheSin- commented Jun 6, 2023

so build is failing for python 3.9, couldn't even test to upgrade back to using pkg_type over top of type_raw

@dmacks
Copy link
Member

dmacks commented Jun 6, 2023

Googling around, it might be because docutils/ contains a python module 'io' that masks a python-core module of the same name, a core module that is also involved in error reporting. So it crashes because it can't even report the error properly. The masking is because PYTHONPATH contains 'docutils'. I think the python scripts and modules import docutils.io when they want that and io when they want the core one. Can our .info just use PYTHONPATH=. ?

@dhomeier
Copy link
Contributor Author

dhomeier commented Jun 7, 2023

Yes, absolutely right – apparently forgot to push that change while this was still in review.

@dak180
Copy link
Member

dak180 commented Jun 7, 2023

dpkg-deb: building package 'docutils-py27' in '/opt/sw/fink/12.0/remote/dak180/binary-darwin-arm64/docutils-py27_0.18.1-1_darwin-arm64.deb'.
Removing runtime build-lock...
Removing build-lock package...
/opt/sw/bin/dpkg-lockwait -r fink-buildlock-docutils-py27-0.18.1-1
(Reading database ... 271941 files and directories currently installed.)
Removing fink-buildlock-docutils-py27-0.18.1-1 (2023.06.06-21.41.08) ...
Reading buildlock packages...
/opt/sw/bin/dpkg-lockwait -i /opt/sw/fink/dists/remote/dak180/binary-darwin-arm64/docutils-py27_0.18.1-1_darwin-arm64.deb
(Reading database ... 271940 files and directories currently installed.)
Preparing to unpack .../docutils-py27_0.18.1-1_darwin-arm64.deb ...
Unpacking docutils-py27 (0.18.1-1) over (0.14-1) ...
Setting up docutils-py27 (0.18.1-1) ...
update-alternatives: error: alternative rst2latex can't be slave of rst2html: it is a master alternative
dpkg: error processing package docutils-py27 (--install):
 installed docutils-py27 package post-installation script subprocess returned error exit status 2
Processing triggers for dpkg-base-files (0.4-1) ...
Clearing dependency_libs of .la files in /opt/sw/lib ...
Errors were encountered while processing:
 docutils-py27
### execution of /opt/sw/bin/dpkg-lockwait failed, exit code 1
Updating the list of locally available binary packages.
Scanning dists/remote/dak180/binary-darwin-arm64
New package: dists/remote/dak180/binary-darwin-arm64/docutils-py27_0.18.1-1_darwin-arm64.deb
Failed: can't install package docutils-py27-0.18.1-1
Making BUGS.html
Making COPYING.html
COPYING.txt:102: (WARNING/2) Inline emphasis start-string without end-string.
Making FAQ.html
Making HISTORY.html
HISTORY.txt:205: (ERROR/3) Unknown target name: "image_loading".
Making README.html
Making RELEASE-NOTES.html
Making THANKS.html
Making manpage-URUE.html
Making math.html
Making mwe-test.html

cd docs
for i in `find . -name '*.txt'` ; do
  destdir="/opt/sw/src/fink.build/root-docutils-py39-0.19-1/opt/sw/share/doc/docutils-py39/html/"${i%/*}
  htmldoc=${i%txt}html
  echo "Making $htmldoc"
  PYTHONPATH=../opt/sw/bin/python3.9 ../tools/rst2html.py $i $htmldoc
  mkdir -p "$destdir"
  cp $htmldoc $destdir
done
find . -name '*.txt'
Making ./ref/doctree.html
Traceback (most recent call last):
  File "/opt/sw/src/fink.build/docutils-py39-0.19-1/docutils-0.19/docs/../tools/rst2html.py", line 17, in <module>
    from docutils.core import publish_cmdline, default_description
ModuleNotFoundError: No module named 'docutils'
### execution of /tmp/fink.JFjCP failed, exit code 1
### execution of /tmp/fink.BS4Dg failed, exit code 1
Removing runtime build-lock...
Removing build-lock package...
/opt/sw/bin/dpkg-lockwait -r fink-buildlock-docutils-py39-0.19-1
(Reading database ... 272366 files and directories currently installed.)
Removing fink-buildlock-docutils-py39-0.19-1 (2023.06.06-21.47.03) ...
Failed: phase installing: docutils-py39-0.19-1 failed
Package manager version: 0.45.99.git
Distribution version: selfupdate-rsync Tue Jun  6 15:19:10 2023, 12.0, arm64
Trees: local/main remote/dak180 stable/main local/injected
Xcode.app: 14.2
Xcode command-line tools: 14.2.0.0.1.1668646533
Max. Fink build jobs:  1

@dhomeier
Copy link
Contributor Author

dhomeier commented Jun 7, 2023

Doh, should not play around with the suggestions menu at this time of night!

@dak180
Copy link
Member

dak180 commented Jun 7, 2023

The following package will be installed or updated:
 docutils-py27
Reading buildlock packages...
/opt/sw/bin/dpkg-lockwait -i /opt/sw/fink/dists/remote/dak180/binary-darwin-arm64/docutils-py27_0.18.1-1_darwin-arm64.deb
(Reading database ... 271940 files and directories currently installed.)
Preparing to unpack .../docutils-py27_0.18.1-1_darwin-arm64.deb ...
Unpacking docutils-py27 (0.18.1-1) over (0.14-1) ...
Setting up docutils-py27 (0.18.1-1) ...
update-alternatives: error: alternative rst2latex can't be slave of rst2html: it is a master alternative
dpkg: error processing package docutils-py27 (--install):
 installed docutils-py27 package post-installation script subprocess returned error exit status 2
Processing triggers for dpkg-base-files (0.4-1) ...
Clearing dependency_libs of .la files in /opt/sw/lib ...
Errors were encountered while processing:
 docutils-py27
### execution of /opt/sw/bin/dpkg-lockwait failed, exit code 1
Failed: can't install package docutils-py27-0.18.1-1

@dhomeier py39 worked though.

@dak180 dak180 self-requested a review June 7, 2023 05:24
@dhomeier
Copy link
Contributor Author

dhomeier commented Jun 7, 2023

update-alternatives: error: alternative rst2latex can't be slave of rst2html: it is a master alternative

Hmm, apparently changing the naming scheme is not causing problems here, but the master-slave setup does.
@nieder, I tried to get around this by always removing any old links first in the PostInstScript of the new package, similar to the ipython-py scripts as

PostInstScript: <<
  # Remove the old scripts if they exist
  for i in rst2html rst2latex rst2man rst2odt rst2odt_prepstyles rst2pseudoxml rst2s5 rst2xetex rst2xml rstpep2html; do
    update-alternatives --auto $i
    update-alternatives --remove $i %p/bin/$i || echo "skipping $i"
  done

but this will fail in turn when updating the new package version again, as then there is only the master alternative rst2html registered:

update-alternatives: error: no alternatives for rst2latex
dpkg: error processing package docutils-py38 (--install):
installed docutils-py38 package post-installation script subprocess returned error exit status 2

and even with the || redirection on error, the script is immediately exiting at this point.
I have not found any way to probe the existence of an alternative without getting an exit 2 if it does not exist, so I can't think of a failsafe way of updating this (and things are complicated further when e.g. updating docutils-py39 over an older installed docutils-py38 version).

I am proposing, in the last commit, to switch to install all other scripts as slaves to rst2html, thus simplifying PreRmScript. This would certainly need some more testing, though.

I am out of my update-alternatives wisdom here, so unless someone still has an idea, I'd revert this change and continue to install all links independently.

@nieder
Copy link
Member

nieder commented Jun 8, 2023

I added --verbose to all the u-a calls in the new .info to track what's going on.
Install old version of py310, then install new version:

$ sudo dpkg -i /sw/fink/debs/docutils-py310_0.14-1_darwin-x86_64.deb 
Selecting previously deselected package docutils-py310.
(Reading database ... 728077 files and directories currently installed.)
Unpacking docutils-py310 (from .../docutils-py310_0.14-1_darwin-x86_64.deb) ...
Setting up docutils-py310 (0.14-1) ...
$ fink install docutils-py310 (this is version 0.19)
Setting up docutils-py310 (0.19-1) ...
Checking available versions of docutils, updating links in /sw/etc/alternatives ...
(You may modify the symlinks there yourself if desired - see `man ln'.)
Updating docutils (/sw/bin/docutils) to point to /sw/bin/docutils-py310.
Updating docutils-py (/sw/bin/docutils.py) to point to /sw/bin/docutils-py310.
Alternative for rst2html points to /sw/bin/rst2html-3.10.py - which wasn't found.  Removing from list of alternatives.
/sw/etc/alternatives/rst2html has been changed (manually or by a script).
Switching to manual updates only.
Checking available versions of rst2html, updating links in /sw/etc/alternatives ...
(You may modify the symlinks there yourself if desired - see `man ln'.)
Automatic updates of /sw/etc/alternatives/rst2html are disabled, leaving it alone.
To return to automatic updates use `update-alternatives --auto rst2html'.
Alternative for rst2latex points to /sw/bin/rst2latex-3.10.py - which wasn't found.  Removing from list of alternatives.
/sw/etc/alternatives/rst2latex has been changed (manually or by a script).
...

Then

$ file /sw/etc/alternatives/rst*
/sw/etc/alternatives/rst2html:              broken symbolic link to /sw/bin/rst2html-3.10.py
/sw/etc/alternatives/rst2html-py:           broken symbolic link to /sw/bin/rst2html-3.10.py
/sw/etc/alternatives/rst2latex:             broken symbolic link to /sw/bin/rst2latex-3.10.py

I think the new docutils-py is going to need a PreInstscript to check for and update-alternatives remove any instances of the old naming scheme (see bundler-rb.info), then proceed with the normal u-a in PostInstScript.

@nieder
Copy link
Member

nieder commented Jun 8, 2023

Oops. missed the previous comment by @dhomeier . I still need to read through it.

@nieder
Copy link
Member

nieder commented Jun 8, 2023

Something like this:

PreInstscript: <<
  # naming scheme changed between 0.14 and 0.18. So we need to make sure
  # we cleanly remove previous u-a scheme here. We can then use
  # PostInstScript as usual to set things up again.
  for i in rst2html rst2latex rst2man rst2odt rst2odt_prepstyles rst2pseudoxml rst2s5 rst2xetex rst2xml rstpep2html; do
   if [ -f %p/bin/%i-%type_raw[python].py ]; do
      echo "Removing u-a for $i from previous version of docutils-py."
      update-alternatives --verbose --auto $i
      update-alternatives --verbose --remove $i %p/bin/$i-%type_raw[python].py
      #update-alternatives --verbose --remove $i-py %p/bin/$i-%type_raw[python].py
    done
  echo "Cleaned up all docutils files from previous release."
  echo ""
<<

PostInstScript: <<
  for i in docutils rst2html rst2latex rst2man rst2odt rst2odt_prepstyles rst2pseudoxml rst2s5 rst2xetex rst2xml rstpep2html; do
    #                                       link         name  path (source)                 priority
    update-alternatives --verbose --install %p/bin/$i    $i    %p/bin/$i-py%type_pkg[python] %type_pkg[python] \
                                  --slave   %p/bin/$i.py $i-py %p/bin/$i-py%type_pkg[python]
  done
<<

PreRmScript: <<
  if [ $1 != "upgrade" ]; then
    for i in docutils rst2html rst2latex rst2man rst2odt rst2odt_prepstyles rst2pseudoxml rst2s5 rst2xetex rst2xml rstpep2html; do
      update-alternatives --verbose --remove $i %p/bin/$i-py%type_pkg[python]
      #update-alternatives --verbose --remove $i-py %p/bin/$i-py%type_pkg[python]
    done
  fi
<<

This worked for me in updating docutils-py310 cleanly from 0.14, with the correct final symlinks. Have not tried it with a different lower py3x going to py310 yet.
Explicitly removing the $i-py files from the original is not needed because they're slaves to the $i, so they'll get pulled automatically.

@TheSin-
Copy link
Member

TheSin- commented Jun 8, 2023

That looks sane to me

Once this is updated and it's confirmed by @dak180, I'll commit it. Unless @dmacks would like to confirm the upgrade path as well? The more the better.

@dhomeier
Copy link
Contributor Author

dhomeier commented Jun 8, 2023

if [ -f %p/bin/%i-%type_raw[python].py ]; do
echo "Removing u-a for $i from previous version of docutils-py."

I almost tried something like that, but thought at that point the actual files from the previous package version would already have been removed...

Have not tried it with a different lower py3x going to py310 yet.

I am afraid that is the critical point – there will be no %p/bin/%i-3.10.py in that case, so should it look for %p/etc/alternatives/%i or just ignore the other py-version?

  • I'd probably have to get to my 10.14 system to check against an installed 0.14 version.

Explicitly removing the $i-pie files from the original is not needed because they're slaves to the $i, so they'll get pulled automatically.

Actually should suffice to just remove rst2html or docutils, whichever is the master?
O or

@nieder
Copy link
Member

nieder commented Jun 8, 2023

if [ -f %p/bin/%i-%type_raw[python].py ]; do
echo "Removing u-a for $i from previous version of docutils-py."

I almost tried something like that, but thought at that point the actual files from the previous package version would already have been removed...

The existing docutils-py.info only removes the symlinks when it's not an upgrade, so we need to be explicit in the new PreInstScript that we don't care what the old version did, we really want to delete the old symlinks.

if [ $1 != "upgrade" ]; then
for i in rst2html rst2latex rst2man rst2odt rst2odt_prepstyles rst2pseudoxml rst2s5 rst2xetex rst2xml rstpep2html; do
update-alternatives --remove $i %p/bin/$i-%type_raw[python].py

Actually should suffice to just remove rst2html or docutils, whichever is the master?

I just noticed that 0.18.1 and 0.19 here have different u-a schemas.
0.14: Each file is a master.
0.18.1: rst2html is the master. All others are slaves (I agree with this transition)
0.19: each file is a master.

Do we want 0.19 to have 1 master w/ many slaves as well? That should be easier for maintenance, but what happens if the file list changes (eg a slave program gets removed upstream)?

@dhomeier
Copy link
Contributor Author

dhomeier commented Jun 8, 2023

Do we want 0.19 to have 1 master w/ many slaves as well? That should be easier for maintenance, but what happens if the file list changes (eg a slave program gets removed upstream)?

Yes, that was actually the idea; I just had not pushed those changes in 0.19 (nor the ones for 0.20.0, which is out in the meantime). And 0.19+ has a docutils binary, so it seemed more sensible to make that the master, but between
0.18.1 master rst2html + slaves
0.19 master docutils + slaves
0.20.0 master docutils + slaves
upgrading works smoothly; just need to make sure PreInstScript does not break on trying to remove the other 0.14 masters.

@dak180
Copy link
Member

dak180 commented Jun 12, 2023

@dhomeier as soon as you have commits I can test them.

@dak180
Copy link
Member

dak180 commented Jun 19, 2023

@dhomeier are there any more commits that you have not yet pushed?

@nieder
Copy link
Member

nieder commented Jan 16, 2024

@dhomeier ping on commits you said you had to make 0.19 (and 0.20?) a master docutils + slaves

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug new upstream Package has an updated upstream version

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docutils-py-0.14 doesn't build on macOS12

6 participants