Codestin Search App

Micky774 · 2023-05-16T18:06:55Z

Reference Issues/PRs

Towards #24686

What does this implement/fix? Explain your changes.

Each change has been separately reviewed (see #24686 for details).

Edit: due to git shenanigans, some changes needed to be made within this PR. The novel changes included w/ this PR are:

Replaced cnp.*_t typing with *_t from _typedefs.pxd
Replaced *.shape[0] pattern with len(*) for ndarray objects
Trimmed unused variables (thanks to Cython linting pre-commit)

Any other comments?

cc: @thomasjpfan @jjerphan @glemaitre

…#22616) Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Julien Jerphanion <[email protected]> Co-authored-by: Guillaume Lemaitre <[email protected]>

Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Julien Jerphanion <[email protected]>

…ikit-learn#25538) Co-authored-by: Julien Jerphanion <[email protected]>

…cikit-learn#24698) Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Julien Jerphanion <[email protected]>

Co-authored-by: Julien Jerphanion <[email protected]> Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Guillaume Lemaitre <[email protected]>

…n#25768)

Co-authored-by: Julien Jerphanion <[email protected]>

Co-authored-by: Thomas J. Fan <[email protected]>

jjerphan · 2023-05-16T18:44:33Z

Since all PRs target the branch of this PR have been reviewed, merged, and are referenced in the issue, I thinking making the CI pass is sufficient to accept this PR.

What do you think?

Micky774 · 2023-05-16T21:00:25Z

I am going to mark it as a draft for now to resolve CI issues (it seems some changes weren't carried over due to git shenanigans) but will reopen once those are addressed.

Micky774 · 2023-05-21T00:41:27Z

Should be good for review now @jjerphan @thomasjpfan!

Since all PRs target the branch of this PR have been reviewed, merged, and are referenced in the issue, I thinking making the CI pass is sufficient to accept this PR.

There are some not-yet-reviewed changes. For your convenience, here are the novel changes (all of which are included in these five commits)

These changes are comprised of:

Deletion of submodule-level setup.py files corresponding to old build strategy
Update to setup.py for building _hdbscan submodule
Sphinx doc appeasement in clustering.rst, plot_hdbscan.py, and hdbscan.py
Updated sklearn/cluster/_hierarchical_fast.pyx::UnionFind to avoid redeclaration of attributes and use new typings
Updated shape semantics in Cython files (len for lists/sets, .shape[i] for memoryviews, PyArray_SHAPE for ndarray objects)

jjerphan

LGTM.

Thank you for bringing HDBSCAN in scikit-learn, @Micky774!

Here are just minor comments.

jjerphan · 2023-05-22T22:14:36Z

I think it is reasonable to wait for a third reviewer to approve this PR before merging it.

Micky774 · 2023-05-22T22:24:31Z

cc: @scikit-learn/core-devs in case anyone would like to donate a third review :)

adrinjalali

The copyright here: https://github.com/scikit-learn-contrib/hdbscan/blob/master/LICENSE isn't really BSD, and doesn't allow to just use it, to me it seems we need a copy of that file here? Unless that copyright is changed or we somehow get a written permission from all contributors to that package.

I left a few comments just to make a few things more readable, otherwise I'm happy to merge.

Micky774 · 2023-05-31T12:56:27Z

@adrinjalali Should be ready for a second-pass :)

adrinjalali

PRETTY KEWL!!!

jjerphan · 2023-05-31T14:04:23Z

Thank you for working on integrating HDBSCAN in scikit-learn, @Micky774! 🎉

Micky774 · 2023-05-31T14:12:08Z

Thank you to everyone for all your help with this! This was a massive collaboration and I greatly appreciate your persistence and patience with review and API discussions <3

@lmcinnes Wanted to ping you as well! Thanks again to you and your team for the fantastic original implementation.

The first major hurdle is done, now to slowly reintroduce some functionality (e.g. boruvka), optimize, and extend support (e.g. float32 data).

…arn#26385) Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Julien Jerphanion <[email protected]> Co-authored-by: Guillaume Lemaitre <[email protected]>

Self-contained review-harness/: diff_manifest.py, contract_lint.py, fidelity_audit.py, run.sh (deterministic state helper), prompts, and the prepared inputs for reviewing scikit-learn#26385 (PLAN.md, DIFF_MANIFEST.md, hunks/, head_files/ snapshot, and per-theme cloud prompts). Co-Authored-By: Claude Opus 4.8 <[email protected]>

Micky774 and others added 11 commits May 16, 2023 12:35

Include HDBSCAN as a sub-module for sklearn.cluster (scikit-learn…

bd04bda

…#22616) Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Julien Jerphanion <[email protected]> Co-authored-by: Guillaume Lemaitre <[email protected]>

CLN Cleaned cluster/_hdbscan/_linkage.pyx (scikit-learn#24857)

488fb1f

Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Julien Jerphanion <[email protected]>

DOC Adds HDBSCAN.dbscan_clustering section to plot_hdbscan.py (sc…

df7cf3a

…ikit-learn#25538) Co-authored-by: Julien Jerphanion <[email protected]>

ENH Extends outlier encoding scheme to HDBSCAN.dbscan_clustering (s…

800c2a6

…cikit-learn#24698) Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Julien Jerphanion <[email protected]>

CLN Cleaned cluster/_hdbscan/_reachability.pyx (scikit-learn#24701)

29c53b7

Co-authored-by: Julien Jerphanion <[email protected]> Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Guillaume Lemaitre <[email protected]>

CLN Update cluster/_hdbscan/_tree.pyx style and syntax (scikit-lear…

70ae320

…n#25768)

CLN Cleaned TreeUnionFind in _hdbscan/_tree.pyx (scikit-learn#25827)

a6e061e

Co-authored-by: Julien Jerphanion <[email protected]>

ENH _hdbscan HIERARCHY data type introduction (scikit-learn#25826)

d013d38

Co-authored-by: Julien Jerphanion <[email protected]>

CLN BFS Style Improvement (scikit-learn#26096)

792833a

Co-authored-by: Thomas J. Fan <[email protected]>

CLN _hdbscan/_tree.pyx algorithms refactor (scikit-learn#26011)

c02c455

CLN HDBSCAN _tree.pyx::do_labelling refactor (scikit-learn#26101)

607dea8

github-actions Bot added cython module:cluster module:utils labels May 16, 2023

Micky774 added 2 commits May 16, 2023 14:07

Updated changelog

792485a

Updated typing

0661984

Micky774 marked this pull request as draft May 16, 2023 20:59

Updated typing, moved away from deprecated numpy API

4da655c

thomasjpfan reviewed May 19, 2023

View reviewed changes

Comment thread sklearn/cluster/_hdbscan/_linkage.pyx Outdated

Micky774 added 4 commits May 18, 2023 21:06

Updated shape semantics based on underlying data structure

7215ff0

Doc fixes

861b143

Updated shape semantics in remaining Cython code

87a43e5

Merge branch 'main' into hdbscan_sync

15513df

Micky774 marked this pull request as ready for review May 21, 2023 00:41

thomasjpfan approved these changes May 22, 2023

View reviewed changes

jjerphan approved these changes May 22, 2023

View reviewed changes

Comment thread sklearn/cluster/_hdbscan/_linkage.pyx Outdated

Comment thread sklearn/cluster/_hdbscan/_tree.pxd Outdated

Comment thread sklearn/cluster/_hierarchical_fast.pxd Outdated

Use cimport for PyArrayObject and add noexcept for UnionFind

3d46b14

jjerphan reviewed May 22, 2023

View reviewed changes

Comment thread sklearn/cluster/_hdbscan/_linkage.pyx Outdated

Updated cimport strategy

495e21f

adrinjalali reviewed May 23, 2023

View reviewed changes

Micky774 added 4 commits May 24, 2023 11:14

Added copy of license

b4bdfbd

Updated documentation of helper functions/methods

3e9ed2e

Improved testing documentation

fe11344

Updated documentation to explicitly include example

04c8b69

adrinjalali approved these changes May 31, 2023

View reviewed changes

adrinjalali merged commit 415b11a into scikit-learn:main May 31, 2023

Micky774 deleted the hdbscan_sync branch May 31, 2023 14:05

Micky774 mentioned this pull request May 31, 2023

add HDBSCAN #14331

Closed

Uh oh!

Uh oh!

Conversation

Micky774 commented May 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

jjerphan commented May 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Micky774 commented May 16, 2023

Uh oh!

Uh oh!

Micky774 commented May 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjerphan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjerphan commented May 22, 2023

Uh oh!

Micky774 commented May 22, 2023

Uh oh!

adrinjalali left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Micky774 commented May 31, 2023

Uh oh!

adrinjalali left a comment

Choose a reason for hiding this comment

Uh oh!

jjerphan commented May 31, 2023

Uh oh!

Micky774 commented May 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Micky774 commented May 16, 2023 •

edited

Loading

jjerphan commented May 16, 2023 •

edited

Loading

Micky774 commented May 21, 2023 •

edited

Loading