Thanks to visit codestin.com
Credit goes to github.com

Skip to content

permutation inference performance using numba #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sjsrey merged 84 commits into from
Jul 5, 2020
Merged

permutation inference performance using numba #116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
e6e9571
WIP: check numba for moran
ljwolf Apr 24, 2020
3006f0e
WIP: keep iterating on algorithmic correctness
ljwolf Apr 24, 2020
2b4a464
inline for performance
ljwolf Apr 27, 2020
fadc325
add jitting
ljwolf Apr 27, 2020
e5abd15
explore ndim broadcasting for permutations
ljwolf Apr 27, 2020
d7e2186
finalize the neighbors_perm strategy
ljwolf Apr 29, 2020
decc9b4
Add neighbors_perm_plus and start of hooking up to Moran_Local. Unfin…
darribas Apr 29, 2020
af6c782
Add notebook to track benchmarks
darribas Apr 29, 2020
998ebd2
Merge conflicts
darribas Apr 29, 2020
fa18c5c
Hook up numba randomisation with Moran_Local; add illustration in not…
darribas Apr 29, 2020
0b59600
Merge branch 'moran-perf' of github.com:darribas/esda into moran-perf
ljwolf May 1, 2020
42c664e
clean up perf optimizations and add dani terminology
ljwolf May 1, 2020
f573d27
add @sjsrey optimizations
ljwolf May 1, 2020
27ec37a
rescale lisas
ljwolf May 1, 2020
f84b393
Correct but slow numba implementation
darribas May 2, 2020
831e6ac
Correct and performant implementation using numba
darribas May 2, 2020
02702fb
Make neighbors_perm_plus run steps only to be sent to workers. Works …
darribas May 5, 2020
ad322ce
Add numba-based parallel implementation + checks added on notebook
darribas May 6, 2020
3856eef
Patch parallel so it works. Add tests. No good
darribas May 6, 2020
c1f7290
Add comparison to pygeoda
darribas May 6, 2020
00da479
Fix update on parallel loop
darribas May 8, 2020
70fd912
Debug parallel implementation w/ prange. Unfinished
darribas May 8, 2020
e0bfc76
Add correct parallel implementation
darribas May 10, 2020
39ebe33
Move parallelism to joblib
darribas May 10, 2020
9abedfb
Add management of n_jobs=-1 in crand_plus
darribas May 10, 2020
958427c
Clean up on parallel_neighbors_perm_plus to leave joblib only
darribas May 10, 2020
739cdab
Add timings on 8core machine
darribas May 10, 2020
efbbe99
Merge branch 'moran_parallel' of https://github.com/darribas/esda int…
darribas May 10, 2020
4312874
use memmapping and generators to reduce memory pressure
ljwolf May 12, 2020
266673d
rearrange to a separate module called crand.py
ljwolf May 12, 2020
4746dbb
add generic local functions
ljwolf May 12, 2020
9f6569c
fix numpy imports
ljwolf May 15, 2020
a2b5a94
[WIP] prepare for the generalization of the statistical functions
ljwolf May 15, 2020
6695b89
fix chunk-relative indexing in preparation for statfunc generalization
ljwolf May 15, 2020
b76d16f
Add stub for workbench notebook
darribas May 16, 2020
44d31eb
Add pygeoda + save_last option to workbench
darribas May 16, 2020
8179a34
Add more options to workbench and timings for tracts
darribas May 16, 2020
7e88e0d
use first column of the input in crand to scale
ljwolf May 22, 2020
e38e93e
Add bivariate scaling and make calc consistent
ljwolf May 22, 2020
ca1b1fd
begin addressing issues with keeping simulations
ljwolf May 22, 2020
107e0d0
Add documentation in crand.py
darribas May 22, 2020
58a74b6
Merge branch 'moran-perf' of github.com:darribas/esda into moran-perf
darribas May 22, 2020
79d55a4
keep univariate z flat
ljwolf May 28, 2020
63b8ac0
Merge pull request #6 from darribas/moran-perf
ljwolf May 29, 2020
69d18de
move imports to implement jit decorator
ljwolf May 29, 2020
12ef70c
deal with no joblib
ljwolf May 29, 2020
a4d0c86
add moran-rate to crand
ljwolf May 29, 2020
73ce4f0
remove old crand codepaths
ljwolf Jun 5, 2020
85ba79c
move local functions into the files for each statistic
ljwolf Jun 5, 2020
1884487
[WIP] add getisord functions
ljwolf Jun 12, 2020
e7a9787
only compute simulation distribution statistics if realizations are r…
ljwolf Jun 12, 2020
8dbba1c
allow local implementations to override scaling
ljwolf Jun 19, 2020
e7d887c
add temporary crand validation tooling
ljwolf Jun 19, 2020
421442e
revert the crand engine changes in getisord
ljwolf Jun 26, 2020
c79b54d
move to keep_simulations=True on by default
ljwolf Jun 26, 2020
42a9dc7
Remove numba parameter in moran.py
darribas Jun 28, 2020
9a988d8
Start of tests for crand
darribas Jun 28, 2020
3097e23
Merge branch 'moran-perf' of https://github.com/ljwolf/esda into mora…
darribas Jun 28, 2020
b9320e8
Update one test
darribas Jul 1, 2020
cf53b7b
Add seed and adjust tests to pass with new randomisation engine
darribas Jul 1, 2020
8151e06
Merge pull request #7 from darribas/moran-perf
ljwolf Jul 1, 2020
2129431
add numba to requirements for testing
ljwolf Jul 3, 2020
fe89816
Merge branch 'master' of github.com:pysal/esda into moran-perf
ljwolf Jul 3, 2020
923fa5d
Merge branch 'moran-perf' of github.com:ljwolf/esda into moran-perf
ljwolf Jul 3, 2020
0467f25
TRAVIS: move numba into matrix, install from conda
ljwolf Jul 3, 2020
0fe9cde
TRAVIS: add numba to can-fail cases
ljwolf Jul 3, 2020
60ff165
TRAVIS: fix if/else for numba
ljwolf Jul 3, 2020
a31cedf
TRAVIS: missed the if/else terminator
ljwolf Jul 3, 2020
a21896f
TRAVIS: try strict channel preference for numba
ljwolf Jul 3, 2020
82ae56b
TRAVIS: try dropping conda-forge because we're using pip anyway
ljwolf Jul 3, 2020
5f0afc2
TRAVIS: forgot the --yes in conda install
ljwolf Jul 3, 2020
93a1445
Add seed argument to be passed to crand and insert in tests
darribas Jul 3, 2020
46a415e
Merge pull request #8 from darribas/moran-perf
ljwolf Jul 3, 2020
6a446a3
Make seed not optional in vec_permutations
darribas Jul 3, 2020
976cb41
Fix fixed-type syntax for seed
darribas Jul 3, 2020
f40ab56
Merge pull request #9 from darribas/moran-perf
ljwolf Jul 3, 2020
3118828
Adding further documentation
darribas Jul 3, 2020
8c85b2c
Clean up notebooks for performance/correctness
darribas Jul 3, 2020
439a684
Merge pull request #10 from darribas/moran-perf
ljwolf Jul 3, 2020
c9b770d
handle the deprecations in unittests
ljwolf Jul 3, 2020
1caa226
fix false discovery rate filtering
ljwolf Jul 3, 2020
e17b8c8
finalize edits for the new seed processing options
ljwolf Jul 3, 2020
277eef4
fix up docscring
ljwolf Jul 3, 2020
f017903
Merge branch 'master' into moran-perf
sjsrey Jul 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 19 additions & 7 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,33 +11,45 @@ python:
- 3.8

env:
- PYSAL_PYPI=true MPLBACKEND="pdf"
- PYSAL_PYPI=false MPLBACKEND="pdf"
- PYSAL_PYPI=true MPLBACKEND="pdf" NUMBA=false
- PYSAL_PYPI=false MPLBACKEND="pdf" NUMBA=false
- PYSAL_PYPI=true MPLBACKEND="pdf" NUMBA=true
- PYSAL_PYPI=false MPLBACKEND="pdf" NUMBA=true

matrix:
allow_failures:
- python: 3.6
env: PYSAL_PYPI=false MPLBACKEND="pdf"
env: PYSAL_PYPI=false MPLBACKEND="pdf" NUMBA=false
- python: 3.7
env: PYSAL_PYPI=false MPLBACKEND="pdf"
env: PYSAL_PYPI=false MPLBACKEND="pdf" NUMBA=false
- python: 3.8
env: PYSAL_PYPI=false MPLBACKEND="pdf"
env: PYSAL_PYPI=false MPLBACKEND="pdf" NUMBA=false
- python: 3.6
env: PYSAL_PYPI=false MPLBACKEND="pdf" NUMBA=true
- python: 3.7
env: PYSAL_PYPI=false MPLBACKEND="pdf" NUMBA=true
- python: 3.8
env: PYSAL_PYPI=false MPLBACKEND="pdf" NUMBA=true


before_install:
- wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
- chmod +x miniconda.sh
- ./miniconda.sh -b -p ./miniconda
- export PATH=`pwd`/miniconda/bin:$PATH
- conda update --yes conda
- conda config --add channels conda-forge
- conda config --set channel_priority strict
- conda create -y -q -n test-env python=$TRAVIS_PYTHON_VERSION
- source activate test-env

install:
- conda install --yes pip
- pip install -r requirements.txt
- pip install -r requirements_tests.txt
- if "$NUMBA"; then
echo 'testing with numba' && conda install --yes "numba>=0.50";
else
echo 'testing without numba';
fi;
- if "$PYSAL_PYPI"; then
echo 'testing pypi libpysal' && pip install libpysal;
else
Expand Down
Loading