Append Unique Suggestions #874

jiayinghsu · 2022-07-29T18:54:06Z

Description

This PR appends labeling suggestions instead of replacing them when the user clicks "Generate Suggestions". To ensure labeling suggestions are unique, we add a filter at the end of each suggestions method that gets rid of any non-unique suggestions. For the stride and random sampling, we create a unique sample space consisting of the frames not yet in the labeling suggestions. This will inevitably slow down suggestion generation (due to looping through already existing suggestions) - a good idea might be to create a cache for labeling suggestions similar to the LabelsDataCache.

Types of changes

Does this address any currently open issues?

"Generate Suggestions" should add to the Labeling Suggestions, not replace them #838

Outside contributors checklist

Review the guidelines for contributing to this repository
Read and sign the CLA and add yourself to the authors list
Make sure you are making a pull request against the develop branch (not main). Also you should start your branch off develop
Add tests that prove your fix is effective or that your feature works
Add necessary documentation (if appropriate)

Thank you for contributing to SLEAP!

❤️

…velop

roomrys

Some superficial changes requested for the first round

roomrys · 2022-07-29T19:28:23Z

environment.yml

-  - cudatoolkit=11.3.1
-  - cudnn=8.2.1
-  - nvidia::cuda-nvcc=11.3
+#  - cudatoolkit=11.3.1
+#  - cudnn=8.2.1
+#  - nvidia::cuda-nvcc=11.3


Remove this file from the commit, we do not want any changes made to environment.yml

roomrys · 2022-07-29T19:54:29Z

sleap/gui/suggestions.py

+        ]
+        unique_idx = set(frame_idxs) - set(prev_idx)
+        suggestions = cls.idx_list_to_frame_list(unique_idx, video)
+        labels.suggestions.append(suggestions)


We do not want to append the suggestions yet as this function only generates suggestions. We append suggestions later by calling Labels.set_suggestions

sleap/sleap/gui/commands.py

Lines 2414 to 2418 in ab63c9d

new_suggestions = VideoFrameSuggestions.suggest(

labels=context.labels, params=params

)

context.labels.set_suggestions(new_suggestions)

sleap/sleap/io/dataset.py

Lines 1475 to 1477 in ab63c9d

def set_suggestions(self, suggestions: List[SuggestionFrame]):

"""Set the suggested frames."""

self.suggestions = suggestions

roomrys · 2022-07-29T20:35:53Z

sleap/gui/suggestions.py

+                # vid_suggestions = list(range(0, video.frames, frame_increment))[
+                #     :per_video
+                # ]


roomrys · 2022-07-29T20:36:03Z

sleap/gui/suggestions.py

                frames_num = per_video
                frames_num = video.frames if (frames_num > video.frames) else frames_num
-                vid_suggestions = random.sample(range(video.frames), frames_num)
+                # vid_suggestions = random.sample(range(video.frames), frames_num)


roomrys · 2022-07-29T20:36:47Z

sleap/gui/suggestions.py

                cls.idx_list_to_frame_list(vid_suggestions, video, group)
            )

+            labels.suggestions.append(suggestions)


Do not append to labels.suggestions here, only return suggestions

roomrys · 2022-07-29T20:39:48Z

sleap/gui/suggestions.py

+
+        # frame_idxs = list(
+        #     map(
+        #         int,
+        #         np.squeeze(
+        #             np.argwhere(displacements - data_min > data_range * threshold)
+        #         ),
+        #     )
+        # )
+
+        # Take away np.squeeze to resolve the case of 0-d array.


roomrys · 2022-07-29T20:55:12Z

environment.yml

-  - cudatoolkit=11.3.1
-  - cudnn=8.2.1
-  - nvidia::cuda-nvcc=11.3
+#  - cudatoolkit=11.3.1
+#  - cudnn=8.2.1
+#  - nvidia::cuda-nvcc=11.3


Remove any changes made to environment.yml

roomrys · 2022-07-29T20:58:55Z

sleap/info/summary.py

        # Shift back by 1 frame so offsets line up with frame index.
-        result[1:] = result[:-1]
+        # result[1:] = result[:-1]


roomrys · 2022-07-29T20:59:34Z

tests/fixtures/videos.py

 TEST_SMALL_ROBOT_SIV_FILE1 = "tests/data/videos/robot1.jpg"
 TEST_SMALL_ROBOT_SIV_FILE2 = "tests/data/videos/robot2.jpg"
-
+TEST_SMALL_ROBOT_VID = "tests/data/videos/test.mp4"


Rename test.mp4 to a more descriptive name describing the contents of the video

roomrys · 2022-07-29T21:00:03Z

tests/gui/test_suggestions.py

+    # track_b = Track(0, "b")
+    # track_c = Track(0, "c")


roomrys

Should be using a set difference '-' instead of symmetric difference ^. Also, we need to account for different videos in image_feature_based_method.

roomrys · 2022-08-16T19:15:18Z

environment_no_cuda.yml

@@ -1,4 +1,4 @@
-name: sleap
+name: swe


Remove these changes, we do not want any changes made to this file.

roomrys · 2022-08-16T20:37:31Z

sleap/gui/suggestions.py

        for video in videos:
+            vid_idx = list(range(video.frames))
+            vid_sugg_idx = sugg_idx_dict[video]
+            print("vid_sugg_idx", vid_sugg_idx)


Remove before merging.

roomrys · 2022-08-16T20:38:00Z

sleap/gui/suggestions.py

+        sugg_idx = [
+            (sugg.video, sugg.frame_idx)
+            for suggestions in labels.suggestions
+            for sugg in suggestions
+        ]
+
+        # Turn list of tuples into dict where first argument in tuple is dict key
+        for vid, frame in sugg_idx:
+            sugg_idx_dict[vid].append(frame)


Can we combine these two operations into a single loop?

roomrys · 2022-08-16T20:58:26Z

sleap/gui/suggestions.py

                frames_num = video.frames if (frames_num > video.frames) else frames_num
-                vid_suggestions = random.sample(range(video.frames), frames_num)
+                if len(unique_idx) == 1:
+                    vid_suggestions = list(unique_idx)
+                else:
+                    vid_suggestions = random.sample(unique_idx, frames_num)


We need to ensure that frames_num <= len(unique_idx) for random.sample(unique_idx, frames_num) to work. Try changing L113 to frames_num = len(unique_idx) if (frames_num > len(unique_idx)) else frames_num.

roomrys · 2022-08-16T21:08:02Z

sleap/gui/suggestions.py

+        sugg_idx_dict = {video: [] for video in videos}
+
+        sugg_idx = [(sugg.video, sugg.frame_idx) for sugg in labels.suggestions]
+
+        # Turn list of tuples into dict where first argument in tuple is dict key
+        for vid, frame in sugg_idx:
+            sugg_idx_dict[vid].append(frame)


Maybe we can do this in a single for loop?

roomrys · 2022-08-16T21:52:35Z

tests/gui/test_suggestions.py

+    instances3.append(
+        PredictedInstance(
+            skeleton=skeleton,
+            score=1,
+            track=track_a,
+            points=dict(
+                a=PredictedPoint(8, 9, score=0.5), b=PredictedPoint(7, 9, score=0.5)
+            ),
+        )
+    )
+
+    instances3.append(
+        PredictedInstance(
+            skeleton=skeleton,
+            score=5,
+            track=track_a,
+            points=dict(
+                a=PredictedPoint(2, 3, score=0.7), b=PredictedPoint(6, 5, score=0.7)
+            ),
+        )
+    )
+
+    instances3.append(
+        PredictedInstance(
+            skeleton=skeleton,
+            score=5,
+            track=track_a,
+            points=dict(
+                a=PredictedPoint(5, 9, score=0.7), b=PredictedPoint(8, 10, score=0.7)
+            ),
+        )
+    )


We could condense this with instances3.extend(<list of PredictedInstances>)

roomrys · 2022-08-16T21:53:12Z

tests/io/test_dataset.py

+    print(labels.labeled_frames)
+    print(labels.videos)
+


roomrys · 2022-08-16T21:53:19Z

tests/io/test_dataset.py

        labels=labels, params=dict(videos=labels.videos, method="sample", per_video=13)
    )
    labels.set_suggestions(suggestions)
+    print(suggestions)


roomrys · 2022-08-16T21:54:02Z

tests/gui/test_suggestions.py

+    # TODO(JX): Double check whether returning one frame each time for image features is expected.
+    # TODO(JX): Double check whether returning no frame during new suggestions for prediction score
+    #  is expected when given 3 frames and one low score instance in each frame.


Remove these if finished

roomrys · 2022-08-16T21:58:34Z

tests/gui/test_suggestions.py

+    if params["method"] == "image features":
+        assert len(suggestions) == 1
+        assert len(new_suggestions) == 1
+    elif params["method"] == "prediction_score":
+        assert len(suggestions) == (
+            (params["per_video"] + 1) * params["instance_limit"]
+        )
+        assert len(new_suggestions) == 0
+    elif params["method"] == "velocity":
+        assert len(suggestions) == 1
+        assert len(new_suggestions) == 0
+    else:
+        assert len(suggestions) == params["per_video"]
+        assert len(new_suggestions) == 1


Can you add some comments as to why we expect these results

codecov · 2022-08-24T21:22:13Z

Codecov Report

Merging #874 (c518e72) into develop (eec8a00) will increase coverage by 0.03%.
The diff coverage is 97.14%.

@@             Coverage Diff             @@
##           develop     #874      +/-   ##
===========================================
+ Coverage    67.59%   67.63%   +0.03%     
===========================================
  Files          130      130              
  Lines        22187    22209      +22     
===========================================
+ Hits         14997    15020      +23     
+ Misses        7190     7189       -1

Impacted Files	Coverage Δ
sleap/gui/commands.py	`58.61% <0.00%> (ø)`
sleap/gui/suggestions.py	`82.67% <100.00%> (+4.17%)`	⬆️
sleap/io/dataset.py	`83.49% <100.00%> (+0.03%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

roomrys

looks good to me

talmo

LGTM!

jiayinghsu and others added 11 commits July 26, 2022 10:58

add test video

b054f97

append unique suggestions in image_feature_based_method

1e6a4f8

modified pytest

53c0314

modified pytest

6fe031f

fixed basic_sample_suggestion_method

a2c8871

fixed image_feature_based_method

6f51bdc

fixed prediction score and velocity method

06ab038

reformat

28dcda6

Merge branch 'develop' of https://github.com/talmolab/sleap into develop

b06fbfc

Merge branch 'develop' of https://github.com/talmolab/sleap into develop

4c8b6e8

Merge branch 'develop' of https://github.com/jiayinghsu/sleap into de…

188edf9

…velop

roomrys requested changes Jul 29, 2022

View reviewed changes

roomrys and others added 9 commits July 29, 2022 15:33

Change workflow to run on pull-requests

072ca3e

Use correct name for pull request in workflows

37d15e0

Add synchronize to workflow

826371a

Merge branch 'develop' of https://github.com/talmolab/sleap into develop

31721d4

review changes

e3267e0

update basic_sample_suggestion_method

7331967

resolved comments

32d92e6

reformatted

d68b0e2

fixed param[2]

dcdf5e9

roomrys requested changes Aug 16, 2022

View reviewed changes

jiayinghsu added 2 commits August 24, 2022 11:22

resolved comments

724f503

modified test_velocity_suggestions

b08cf21

roomrys added 5 commits September 1, 2022 13:52

Merge branch 'develop' of https://github.com/talmolab/sleap into develop

94341fe

Touch-up basic suggestions

8fc9c28

Touch-up image-based suggestion method

cf7023c

Touch-up prediction score suggestions method

68908ac

Touch-up velocity suggestion method

890e32d

Descriptively rename new video fixture

b748df8

roomrys approved these changes Sep 2, 2022

View reviewed changes

Increase test coverage

c518e72

roomrys requested a review from talmo September 6, 2022 16:39

talmo approved these changes Sep 6, 2022

View reviewed changes

roomrys merged commit 4de5213 into talmolab:develop Sep 6, 2022

roomrys mentioned this pull request Sep 11, 2022

SLEAP v1.2.7 #943

Merged

	new_suggestions = VideoFrameSuggestions.suggest(
	labels=context.labels, params=params
	)

	context.labels.set_suggestions(new_suggestions)

	def set_suggestions(self, suggestions: List[SuggestionFrame]):
	"""Set the suggested frames."""
	self.suggestions = suggestions

Uh oh!

Append Unique Suggestions #874

Append Unique Suggestions #874

Uh oh!

Conversation

jiayinghsu commented Jul 29, 2022 • edited by roomrys Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Does this address any currently open issues?

Outside contributors checklist

Thank you for contributing to SLEAP!

Uh oh!

roomrys left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roomrys left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Aug 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

roomrys left a comment

Choose a reason for hiding this comment

Uh oh!

talmo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jiayinghsu commented Jul 29, 2022 •

edited by roomrys

Loading

codecov bot commented Aug 24, 2022 •

edited

Loading