Thanks to visit codestin.com
Credit goes to github.com

Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
dec4b67
[cd build] Build wheels for arm64 on circleci.
cmarmo Aug 8, 2021
5137014
Remove travis.
cmarmo Aug 10, 2021
4c030e0
Merge branch 'main' into circle-cd-build-arm
Aug 20, 2021
8ee5826
Move condition in circleci config.
cmarmo Aug 20, 2021
1e69d03
Fix parameter default.
cmarmo Aug 20, 2021
a8d2caa
Fix string.
cmarmo Aug 20, 2021
cedadc1
Fix matches condition.
cmarmo Aug 20, 2021
303b4c4
Add wheels.
cmarmo Aug 20, 2021
39925a1
Add script to build wheels
cmarmo Aug 20, 2021
4260426
Move conditions in workflows.
cmarmo Aug 20, 2021
078f273
[cd build] Trig wheels.
cmarmo Aug 20, 2021
9216184
Rename wheel script. Fix pattern.
cmarmo Aug 21, 2021
6d4ee5c
[cd build] Trig wheels.
cmarmo Aug 21, 2021
7cc4fd7
[cd build]Redefine regexp.
cmarmo Aug 21, 2021
94a8c3c
[cd build]Redefine regexp.
cmarmo Aug 21, 2021
c58c685
[cd build] Redefine regexp.
cmarmo Aug 21, 2021
239fbf1
[cd build] Fix version.
cmarmo Aug 21, 2021
0497b50
Move when condition to job.
cmarmo Aug 22, 2021
827dc4e
Fix config.
cmarmo Aug 22, 2021
8ebde47
Move condition back to workflow.
cmarmo Aug 22, 2021
fd2c531
Set parameter.
cmarmo Aug 22, 2021
b39bf63
Remove unuseful definition.
cmarmo Aug 22, 2021
6c9fc0b
Revert.
cmarmo Aug 22, 2021
1b2ca79
Remove condition.
cmarmo Aug 23, 2021
bc0174e
Fix.
cmarmo Aug 23, 2021
eec5805
Fix again.
cmarmo Aug 23, 2021
bcc8906
Fix again.
cmarmo Aug 23, 2021
23a97c3
[cd build] Trig wheels.
cmarmo Aug 23, 2021
8eda6ba
[cd build] Trig wheels.
cmarmo Aug 23, 2021
808ff72
[cd build] Fix again.
cmarmo Aug 23, 2021
0d42150
[cd build] Fix unexpected EOF.
cmarmo Aug 23, 2021
5070a2a
[cd build] Fix unexpected EOF.
cmarmo Aug 23, 2021
c49f39d
Merge branch 'main' into circle-cd-build-arm
cmarmo Sep 2, 2021
ac67f6b
Add the number of new wheels. Add wheel tests. Address comments.
cmarmo Sep 2, 2021
1a89340
[cd build] Trig wheel build.
cmarmo Sep 2, 2021
4f63c11
Update build_tools/github/check_wheels.py
ogrisel Sep 3, 2021
2da7ae1
MAINT setting up staging area upload step [cd build]
ogrisel Sep 6, 2021
f1a0933
Try to fix the upload_anaconda.sh call
ogrisel Sep 6, 2021
1e5a058
[cd build]
ogrisel Sep 6, 2021
078fb19
Define in circle ci to be homegenous with github actions [cd build]
ogrisel Sep 6, 2021
09df028
bash syntax error [cd build]
ogrisel Sep 6, 2021
92c07da
Reenable full linux/arm64 wheel [cd build]ing
ogrisel Sep 6, 2021
1bc8ab7
Try to expaned HOME variable in circleci config [cd build]
ogrisel Sep 7, 2021
a5169db
This PATH and CONDA vars should be defined correctly [cd build]
ogrisel Sep 7, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,28 @@ jobs:
paths:
- ~/scikit_learn_data

linux-arm64-wheels:
machine:
image: ubuntu-2004:202101-01
resource_class: arm.medium
environment:
- CIBW_BUILD_VERBOSITY: 1
- CIBW_TEST_REQUIRES: pytest pandas threadpoolctl
- CIBW_TEST_COMMAND: bash {project}/build_tools/github/test_wheels.sh
steps:
- checkout
- run:
command: |
if [[ "$(git log --format=%B -n 1 $CIRCLE_SHA1)" =~ \[cd\ build\] ]]; then
bash build_tools/circle/checkout_merge_commit.sh
bash build_tools/circle/build_wheels_arm.sh
fi
- store_artifacts:
path: wheelhouse
destination: wheelhouse
- run: bash build_tools/github/upload_anaconda.sh


deploy:
docker:
- image: circleci/python:3.7
Expand All @@ -177,7 +199,6 @@ jobs:
fi

workflows:
version: 2
build-doc-and-deploy:
jobs:
- lint
Expand Down Expand Up @@ -208,3 +229,4 @@ workflows:
linux-arm64:
jobs:
- linux-arm64
- linux-arm64-wheels
73 changes: 0 additions & 73 deletions .travis.yml

This file was deleted.

5 changes: 1 addition & 4 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,13 +1,10 @@
.. -*- mode: rst -*-

|Azure|_ |Travis|_ |Codecov|_ |CircleCI|_ |Nightly wheels|_ |Black|_ |PythonVersion|_ |PyPi|_ |DOI|_
|Azure|_ |Codecov|_ |CircleCI|_ |Nightly wheels|_ |Black|_ |PythonVersion|_ |PyPi|_ |DOI|_

.. |Azure| image:: https://dev.azure.com/scikit-learn/scikit-learn/_apis/build/status/scikit-learn.scikit-learn?branchName=main
.. _Azure: https://dev.azure.com/scikit-learn/scikit-learn/_build/latest?definitionId=1&branchName=main

.. |Travis| image:: https://api.travis-ci.com/scikit-learn/scikit-learn.svg?branch=main
.. _Travis: https://travis-ci.com/scikit-learn/scikit-learn

.. |Codecov| image:: https://codecov.io/gh/scikit-learn/scikit-learn/branch/main/graph/badge.svg?token=Pk8G9gg3y9
.. _Codecov: https://codecov.io/gh/scikit-learn/scikit-learn

Expand Down
24 changes: 24 additions & 0 deletions build_tools/circle/build_wheels_arm.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/bin/bash

set -e
set -x

sudo apt-get update
sudo apt-get install -y wget

# Install Python 3.8 with conda from miniforge because this is
# the simplest way to get a working anaconda-client for the wheel
# upload step to the staging area on the anaconda.org service.
export CONDA=$HOME/mambaforge
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh
bash Mambaforge-$(uname)-$(uname -m).sh -b -f -p $CONDA
echo 'export CONDA=$CONDA' >> $BASH_ENV # to be reused in a later step
export PATH=$CONDA/bin:$PATH
mamba update --yes conda
pip install --upgrade pip
for pyversion in 37 38 39; do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am afraid that building the ARM wheels sequentially will slow down the release process. Actually, for 3 python, circle ci seems to be able to complete in 25m which is not too bad. Maybe this is because of ccache?

Still we might want to have one CI job per Python version and leverage Circle CI parallelism https://circleci.com/docs/2.0/parallelism-faster-jobs/. Although one would have to check how many concurrent workers are offered by default in the free Circle Ci account.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I am confused when reading the logs of the cibuildwheel: it seems that it's always using Python3.8 to build all the 3 wheel versions. This means that we will build 3.7 wheels against the Python 3.8 ABI.

I think we need to make sure that each Python version is run in its own circleci job config within an image with the matching Python version: https://circleci.com/docs/2.0/circleci-images/#python (3.7, 3.8 and 3.9).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, according to https://circleci.com/pricing/ I don't think we can benefit from parallelism. So no need to try to get this to work.

However we still need to split the for loop into independent jobs to make sure we use the right Python version for each generated wheel.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to make sure that each Python version is run in its own circleci job config within an image with the matching Python version: https://circleci.com/docs/2.0/circleci-images/#python (3.7, 3.8 and 3.9).

If I understand correctly those customized python images are not available for arm (see the available images for arm documentation), this means to reinstall a conda environment for each wheel on the host image ubuntu-2004:202101-01.

Actually I am confused when reading the logs of the cibuildwheel: it seems that it's always using Python3.8 to build all the 3 wheel versions. This means that we will build 3.7 wheels against the Python 3.8 ABI.

I am confused too then... :) ... I was thinking that the CIBW_BUILD variable was requesting a specific python version in the manylinux image... eg this line shows that python 3.9 is called ... but it is not clear to me if it is during the build of the wheel... perhaps I should also add a bit of verbosity during the WIP state.

Anyway, I understand your concern, let me create the conda envs then.

Copy link
Member

@ogrisel ogrisel Sep 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you are right! For some reason I thought cibuildwheel used docker only for the repair step but it's indeed actually also used for the build step: https://github.com/pypa/cibuildwheel/blob/144733dc60fe0c9c3114e7ef2e4996ad919fec5c/cibuildwheel/linux.py#L186

Too bad that the which python command does not print its output in the log, that would have lifted the ambiguity.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have configured CIBW_BUILD_VERBOSITY: 1 as for the other wheels. Hope this will help.

pywheel=`echo "cp"$pyversion"-manylinux_aarch64"`
export CIBW_BUILD=$pywheel
python -m pip install cibuildwheel
python -m cibuildwheel --output-dir wheelhouse
done
12 changes: 3 additions & 9 deletions build_tools/github/check_wheels.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,12 @@
# 2 windows wheel (win32 + wind_amd64)
n_wheels = 7 * n_python_versions

# plus three more for linux/arm64 on circleci
n_wheels += 3

# plus one more for the sdist
n_wheels += 1

# aarch64 builds from travis
travis_config_path = Path.cwd() / ".travis.yml"
with travis_config_path.open("r") as f:
travis_config = yaml.safe_load(f)

jobs = travis_config["jobs"]["include"]
travis_builds = [j for j in jobs if any("CIBW_BUILD" in env for env in j["env"])]
n_wheels += len(travis_builds)

dist_files = list(Path("dist").glob("**/*"))
n_dist_files = len(dist_files)

Expand Down
24 changes: 15 additions & 9 deletions build_tools/github/upload_anaconda.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@
set -e
set -x

# Install Python 3.8 because of a bug with Python 3.9
export PATH=$CONDA/bin:$PATH
conda create -n upload -y python=3.8
source activate upload
conda install -y anaconda-client
which anaconda

# TODO: how to detect nightly builds for ARM on Circle CI?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can use "$GITHUB_EVENT_NAME" == "schedule" or "$CIRCLE_BRANCH" == "main"?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking on something like this. We might want to design a uniform strategy that would work for any CI. Let's keep the nightly build config for a follow-up PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

if [ "$GITHUB_EVENT_NAME" == "schedule" ]; then
ANACONDA_ORG="scipy-wheels-nightly"
ANACONDA_TOKEN="$SCIKIT_LEARN_NIGHTLY_UPLOAD_TOKEN"
Expand All @@ -11,12 +19,10 @@ else
ANACONDA_TOKEN="$SCIKIT_LEARN_STAGING_UPLOAD_TOKEN"
fi

# Install Python 3.8 because of a bug with Python 3.9
export PATH=$CONDA/bin:$PATH
conda create -n upload -y python=3.8
source activate upload
conda install -y anaconda-client

# Force a replacement if the remote file already exists
anaconda -t $ANACONDA_TOKEN upload --force -u $ANACONDA_ORG dist/artifact/*
echo "Index: https://pypi.anaconda.org/$ANACONDA_ORG/simple"
if [ "$ANACONDA_TOKEN" == "" ]; then
echo "No upload token defined to upload to anaconda.org/$ANACONDA_ORG: skipping."
else
# Force a replacement if the remote file already exists
anaconda -t $ANACONDA_TOKEN upload --force -u $ANACONDA_ORG dist/artifact/*
echo "Index: https://pypi.anaconda.org/$ANACONDA_ORG/simple"
fi
35 changes: 0 additions & 35 deletions build_tools/travis/after_success.sh

This file was deleted.

13 changes: 0 additions & 13 deletions build_tools/travis/install.sh

This file was deleted.

66 changes: 0 additions & 66 deletions build_tools/travis/install_main.sh

This file was deleted.

6 changes: 0 additions & 6 deletions build_tools/travis/install_wheels.sh

This file was deleted.

14 changes: 0 additions & 14 deletions build_tools/travis/script.sh

This file was deleted.

8 changes: 0 additions & 8 deletions build_tools/travis/test_docs.sh

This file was deleted.

Loading