Codestin Search App

skinan · 2023-02-16T14:06:52Z

ClearML is a leading MLOps stack that can supercharge dialogues research with its state-of-the-art experiment tracking capability. ClearML: https://clear.ml/

I have added a clearml_handler.py which contains ClearMLHandler, ClearMLStatsHandler, and ClearMLImageHandler class. Basically, clearml can track everything which is tracked by tensorboard including scalars, and debug samples, and can also store models, artifacts, and the console in the ClearML server which can be easily accessed from ClearML WebUI shown as below:

ClearMLStatsHandler and ClearMLImageHandler can be used with Pytorch Trainer just like TensorboardStatsHandler and TensorboardImageHandler.

Use ClearMLStatsHandler() & ClearMLImageHandler(log_dir="./runs/", batch_transform=from_engine(["image", "label"]),output_transform=from_engine(["pred"]),) with any MONAI example to test its functionality.

Also, please let us where should we put further documentation and tutorials regarding this MLOps tool in the MONAI.

…grate-clearml

Nic-Ma

Thanks for the contribution.
Please add some unit tests to cover the feature.

Thanks.

…grate-clearml

skinan · 2023-02-17T06:48:11Z

Working on adding some unittests.

for more information, see https://pre-commit.ci

…grate-clearml

for more information, see https://pre-commit.ci

…grate-clearml

for more information, see https://pre-commit.ci

skinan · 2023-03-08T11:22:35Z

@wyli , @Nic-Ma , @binliunls, is the memory ran out of error on these Tests normal? The failing tests are blocking the PR.

wyli · 2023-03-08T12:33:38Z

@wyli , @Nic-Ma , @binliunls, is the memory ran out of error on these Tests normal? The failing tests are blocking the PR.

yes, that issue is probably introduced by the code changes in this PR. any idea what the root cause is? we can skip the windows test using @skip_if_windows

MONAI/tests/utils.py

Line 247 in b85e2c6

def skip_if_windows(obj):

but preferably we should understand the issue first.

skinan · 2023-03-08T13:24:45Z

@wyli , Could you please clarify what the coverage command is doing here: https://github.com/Project-MONAI/MONAI/actions/runs/4352352552/jobs/7605037274#step:12:18505 . Because, we need to understand why it has triggered the clearml fire binding. As it was not supposed to trigger clearml.

Also, I would request that can you please check monitoring on the runner machine, so we can understand if it truly ran out of memory.

wyli · 2023-03-08T13:38:19Z

@wyli , Could you please clarify what the coverage command is doing here: https://github.com/Project-MONAI/MONAI/actions/runs/4352352552/jobs/7605037274#step:12:18505 . Because, we need to understand why it has triggered the clearml fire binding. As it was not supposed to trigger clearml.

Also, I would request that can you please check monitoring on the runner machine, so we can understand if it truly ran out of memory.

that's testing the bundle command:

python -m monai.bundle run --runner_id evaluating ...

you can run the test locally with:

python -m tests.test_integration_bundle_run

@Nic-Ma @binliunls could you please help the debugging here?

Nic-Ma · 2023-03-08T14:45:24Z

/build

Signed-off-by: skinan <[email protected]>

wyli · 2023-03-09T08:29:18Z

looking at the error messages, it seems the error is from the clearml fire binding on windows.

     component_trace = _Fire(component, args, parsed_flag_args, context, name)\r
  File "C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\lib\\site-packages\\fire\\core.py", line 475, in _Fire\r
    component, remaining_args = _CallAndUpdateTrace(\r
  File "C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\lib\\site-packages\\clearml\\binding\\frameworks\\__init__.py", line 36, in _inner_patch\r
    raise ex\r
  File "C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\lib\\site-packages\\clearml\\binding\\frameworks\\__init__.py", line 34, in _inner_patch\r
    ret = patched_fn(original_fn, *args, **kwargs)\r
  File "C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\lib\\site-packages\\clearml\\binding\\fire_bind.py", line 172, in __CallAndUpdateTrace\r
    PatchFire.__groups, PatchFire.__commands = PatchFire.__get_all_groups_and_commands(\r
  File "C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\lib\\site-packages\\clearml\\binding\\fire_bind.py", line 249, in __get_all_groups_and_commands\r
    PatchFire._commands_sep.join(query_group) + PatchFire._commands_sep if len(query_group) > 0 else ""\r

https://github.com/Project-MONAI/MONAI/actions/runs/4371805968/jobs/7648041295

Signed-off-by: skinan <[email protected]>

I, Victor Sonck <[email protected]>, hereby add my Signed-off-by to this commit: ce0ac23 Signed-off-by: Victor Sonck <[email protected] Signed-off-by: Victor Sonck <[email protected]>

Signed-off-by: skinan <[email protected]>

wyli · 2023-03-12T10:53:14Z

/build

wyli · 2023-03-12T19:59:22Z

the docker test is blocked by the unit test https://github.com/Project-MONAI/MONAI/actions/runs/4397243843/jobs/7700566567 could you please help debug it? @skinan

the test is running on a github runner https://github.com/Project-MONAI/MONAI/actions/runs/4397243843/workflow#L83-L99

skinan · 2023-03-14T06:15:13Z

@wyli , This is weird. We are trying to reproduce it locally, to find out what goes wrong.

wyli · 2023-03-14T12:02:39Z

@wyli , This is weird. We are trying to reproduce it locally, to find out what goes wrong.

thanks, I try to reproduce it on github by manually triggering the test dev...wyli:MONAI:trigger-tests but it works fine now https://github.com/wyli/MONAI/actions/runs/4415391874 I'll monitor this test for the dev branch

**Issue:** fixes #6148 **Previous Pull-request:** #6013 --------- Signed-off-by: skinan <[email protected]>

**Issue:** fixes Project-MONAI#6148 **Previous Pull-request:** Project-MONAI#6013 --------- Signed-off-by: skinan <[email protected]>

### Description Add option to install `clearml` as optional dependency with `pip install monai[clearml]`. All of the docstrings were updated and unit tests added in [PR #6013](#6013). ### Types of changes  - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. --------- Signed-off-by: revital <[email protected]>

skinan added 10 commits February 9, 2023 10:23

ClearML Integration (with Tensorboard)

d4b52d1

Merge branch 'Project-MONAI:dev' into integrate-clearml

a5c02e2

Merge branch 'Project-MONAI:dev' into integrate-clearml

32606b8

Updates on ClearML Integration

2296a34

default_tag

3dbcd27

Resolve type-formatting issues and others

6bcd8c6

Merge branch 'Project-MONAI:dev' into integrate-clearml

76887ab

Merge branch 'integrate-clearml' of github.com:skinan/MONAI into inte…

c457b9e

…grate-clearml

Merge branch 'Project-MONAI:dev' into integrate-clearml

38e4a2a

Merge branch 'Project-MONAI:dev' into integrate-clearml

c5698e1

Nic-Ma requested a review from binliunls February 16, 2023 14:40

Nic-Ma reviewed Feb 17, 2023

View reviewed changes

Comment thread monai/handlers/clearml_handlers.py Outdated

Comment thread monai/handlers/clearml_handlers.py Outdated

skinan added 3 commits February 17, 2023 06:44

Resolved reviews on pull-request

acec1eb

Merge branch 'integrate-clearml' of github.com:skinan/MONAI into inte…

e6c17dc

…grate-clearml

Merge branch 'Project-MONAI:dev' into integrate-clearml

4c62eed

binliunls reviewed Feb 20, 2023

View reviewed changes

Comment thread monai/handlers/clearml_handlers.py Outdated

Comment thread monai/handlers/clearml_handlers.py Outdated

Comment thread monai/handlers/clearml_handlers.py Outdated

Comment thread monai/handlers/clearml_handlers.py

skinan added 2 commits February 24, 2023 17:36

Merge branch 'Project-MONAI:dev' into integrate-clearml

e87a91f

Fixed typehints based pull-request review.

ef42a1c

binliunls reviewed Feb 28, 2023

View reviewed changes

Comment thread monai/handlers/clearml_handlers.py Outdated

skinan and others added 10 commits February 28, 2023 16:34

Formatting, Unit Test added, type hinting fixed

302d81e

[pre-commit.ci] auto fixes from pre-commit.com hooks

1971492

for more information, see https://pre-commit.ci

"output_uri" type hinting fixed

3319530

Merge branch 'integrate-clearml' of github.com:skinan/MONAI into inte…

7a91055

…grate-clearml

type hinting fixed

89fb3c0

[pre-commit.ci] auto fixes from pre-commit.com hooks

96f38fc

for more information, see https://pre-commit.ci

Type hinting

5dd1c09

Merge branch 'integrate-clearml' of github.com:skinan/MONAI into inte…

87078d8

…grate-clearml

[pre-commit.ci] auto fixes from pre-commit.com hooks

87ff034

for more information, see https://pre-commit.ci

Merge branch 'Project-MONAI:dev' into integrate-clearml

9b23176

close clearml tasks

619dbf2

Signed-off-by: skinan <[email protected]>

Nic-Ma reviewed Mar 9, 2023

View reviewed changes

Comment thread tests/test_handler_clearml_stats.py

auto_connect_arg_parse=False

07692df

Signed-off-by: skinan <[email protected]>

skinan added 2 commits March 12, 2023 12:35

Merge branch 'dev' into integrate-clearml

f9a1dea

clearml version updated

c57884a

Signed-off-by: skinan <[email protected]>

wyli reviewed Mar 12, 2023

View reviewed changes

Comment thread monai/handlers/clearml_handlers.py Outdated

skinan and others added 3 commits March 12, 2023 09:42

Change the position of optional import Task

41a5853

Signed-off-by: skinan <[email protected]>

DCO Remediation Commit for Victor Sonck <[email protected]>

453d51e

I, Victor Sonck <[email protected]>, hereby add my Signed-off-by to this commit: ce0ac23 Signed-off-by: Victor Sonck <[email protected] Signed-off-by: Victor Sonck <[email protected]>

type hinting fixed

109499f

Signed-off-by: skinan <[email protected]>

wyli enabled auto-merge (squash) March 12, 2023 10:53

wyli merged commit f754928 into Project-MONAI:dev Mar 12, 2023

skinan deleted the integrate-clearml branch March 12, 2023 16:42

wyli mentioned this pull request Mar 15, 2023

test_handler_clearml_image cannot finish #6148

Closed

skinan restored the integrate-clearml branch March 15, 2023 12:08

skinan mentioned this pull request Mar 21, 2023

[Fix] Solve for ClearML Test issues. #6212

Merged

wyli pushed a commit that referenced this pull request Mar 21, 2023

[Fix] Solve for ClearML Test issues. (#6212)

d57bff6

**Issue:** fixes #6148 **Previous Pull-request:** #6013 --------- Signed-off-by: skinan <[email protected]>

jak0bw pushed a commit to jak0bw/MONAI that referenced this pull request Mar 28, 2023

[Fix] Solve for ClearML Test issues. (Project-MONAI#6212)

a30972a

**Issue:** fixes Project-MONAI#6148 **Previous Pull-request:** Project-MONAI#6013 --------- Signed-off-by: skinan <[email protected]>

jak0bw pushed a commit to jak0bw/MONAI that referenced this pull request Mar 28, 2023

[Fix] Solve for ClearML Test issues. (Project-MONAI#6212)

80d4f63

**Issue:** fixes Project-MONAI#6148 **Previous Pull-request:** Project-MONAI#6013 --------- Signed-off-by: skinan <[email protected]>

pollfly mentioned this pull request Aug 7, 2023

Add ClearML as optional dependency #6827

Merged

4 tasks

Uh oh!

Conversation

skinan commented Feb 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nic-Ma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

skinan commented Feb 17, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skinan commented Mar 8, 2023

Uh oh!

wyli commented Mar 8, 2023

Uh oh!

skinan commented Mar 8, 2023

Uh oh!

wyli commented Mar 8, 2023

Uh oh!

Nic-Ma commented Mar 8, 2023

Uh oh!

Uh oh!

wyli commented Mar 9, 2023

Uh oh!

Uh oh!

wyli commented Mar 12, 2023

Uh oh!

wyli commented Mar 12, 2023

Uh oh!

skinan commented Mar 14, 2023

Uh oh!

wyli commented Mar 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

skinan commented Feb 16, 2023 •

edited

Loading