Upload Pre-trained Models for Fine Tuning. #896

Lucaszw · 2016-07-11T19:57:54Z

Expects original model definitions from #891

IsaacYangSLA · 2016-07-13T22:16:02Z

This will be useful if we later need to store several pre-trained models in/alongside DIGITS.

Lucaszw · 2016-07-13T22:29:00Z

For sure!

jmancewicz · 2016-07-18T17:09:59Z

digits/templates/partials/home/pretrained_model_pane.html

+                </tbody>
+                <tbody ng-if="jobs.length == 0">
+                    <tr>
+                        <td colspan="{[fields.length]}">


This colspan needs to include the displayed optional columns.

Like

{[ colspan = (storage.pretrained_model_fields | filter:show(true)).length + (storage.model_output_fields | filter:show(true)).length;'' ]} <td colspan="{[colspan]}">

If you remove the model_output.fields from the page, remove them from this colspan as well.

Good point! I will fix this in my next commit.

jmancewicz · 2016-07-18T17:34:34Z

What is there a mechanism for adding the models to the pre-trained model list?

Lucaszw · 2016-07-18T19:46:38Z

Thanks Joe!

I added an option in the New Model drop-down for uploading a pretrained model. There you select your weights, model def, and labels file. The data corresponds to the pretrained model job after uploading.

jmancewicz · 2016-07-18T23:12:00Z

Would it make sense to allow it to point to a tgz file, view the table, and have that as an option for uploading models? There might be too much to assume about the role of files, but if it's a model from DIGITS, it should work. Just a thought.

Lucaszw · 2016-07-19T01:01:44Z

That would definitely be a great option to have! I did think think about it, but decided against because of what you mentioned with external models, and also because it expects the original network prototxt file, and this wasn't saved into tar files prior to #891 .

lukeyeager · 2016-07-26T00:05:05Z

also because it expects the original network prototxt file, and this wasn't saved into tar files prior to #891

Can't we just check the info.json file for an "original" file and reject the upload if it doesn't exist?

Also, you've got a merge conflict now that we merged #904.

Lucaszw · 2016-07-27T20:06:23Z

I added the suggested changes with my latest commit. Everything now should be good to go! :)

lukeyeager · 2016-08-01T20:01:23Z

digits/pretrained_model/__init__.py

@@ -0,0 +1,3 @@
+# Copyright (c) 2014-2016, NVIDIA CORPORATION.  All rights reserved.


This should just be "2016" since it's a new file.

lukeyeager · 2016-08-01T20:09:57Z

The file upload doesn't look very good. @jmancewicz figured out how to make this work for all browsers: #325

But I can't figure out how to add it to this PR because you're defining the form in javascript!?

lukeyeager · 2016-08-01T20:55:09Z

I don't think it makes sense to display columns like "accuracy" or "loss" for pretrained models on the home page, right?
Is there any way to edit the job name after the fact?
In a later PR, let's do some validation of the .prototxt before accepting it. It's annoying to get errors like don't set the data_param.source two or three pages after the one where you screwed up. But no need to fix it right away.

Lucaszw · 2016-08-01T23:13:19Z

Thanks @lukeyeager ! I implemented your comments.

lukeyeager · 2016-08-02T17:12:18Z

I tried uploading tarballs from 2 different models and I just get a red "Upload Failed" with no more information.

Lucaszw · 2016-08-02T18:02:28Z

@lukeyeager My bad! The problem should be fixed now. I added some better error messages for upload as well.

lukeyeager · 2016-08-04T23:53:17Z

digits/model/tasks/caffe_train.py


        self.solver_file = CAFFE_SOLVER_FILE
-        self.original_file = CAFFE_ORIGINAL_FILE
+        self.model_file = CAFFE_ORIGINAL_FILE


Can we use network_file instead of model_file here? I try to be consistent in calling the .prototxt file a "network description" and not calling it a model unless it has weights attached to it.

The nomenclature I'd like to migrate to (but don't fully support yet) is:

"network" for a .prototxt file

"model" or "trained model" for a .prototxt and a corresponding .caffemodel file

"training" for a group of models

Oh nevermind. We do the same for Torch. Rats.

However, you will want to be more careful with the upgrade path here.

Bump the pickle version

In __setstate__, upgrade from original_file to model_file appropriately

Lucaszw · 2016-08-05T02:12:08Z

Thanks @lukeyeager ! I made the changes, and tried out the download feature across different model jobs created at different times to test changing from the network name from original_file to model_file .

changed os.rename to shutil.move

lukeyeager · 2016-08-09T17:55:22Z

This still isn't perfect, but it's working well for me and @IsaacYangSLA and we need it to build other functionality on top, so I'm going to merge it as-is.

TODOs:

Don't ask for a labels file?
Deal with rendering issue with upload button on Ubuntu 16.04 + Firefox 48
Make sure uploaded models always show up on the home page without needing refresh (race condition?)

lukeyeager · 2016-08-09T17:55:32Z

Thanks @Lucaszw!

…aining Upload Pre-trained Models for Fine Tuning.

Lucaszw added enhancement caffe UI labels Jul 11, 2016

Lucaszw force-pushed the uploadPretrainedModelForTraining branch 2 times, most recently from a58a8fe to 2f7f75c Compare July 12, 2016 18:12

Lucaszw removed the caffe label Jul 12, 2016

jmancewicz reviewed Jul 18, 2016
View reviewed changes

Lucaszw mentioned this pull request Jul 25, 2016

Convert Training Jobs to Pretrained Model Jobs #932

Merged

Lucaszw force-pushed the uploadPretrainedModelForTraining branch from 2f7f75c to 2abae97 Compare July 27, 2016 20:04

Lucaszw mentioned this pull request Jul 28, 2016

Layer Visualization And Weights for Pretrained Jobs #937

Closed

lukeyeager reviewed Aug 1, 2016
View reviewed changes

lukeyeager self-assigned this Aug 1, 2016

Lucaszw force-pushed the uploadPretrainedModelForTraining branch 2 times, most recently from 7181906 to 7b6fcdb Compare August 1, 2016 22:59

Lucaszw force-pushed the uploadPretrainedModelForTraining branch 2 times, most recently from 30412eb to b16f0fe Compare August 2, 2016 17:52

Lucaszw force-pushed the uploadPretrainedModelForTraining branch from b16f0fe to 1fbf9d7 Compare August 2, 2016 18:01

Lucaszw force-pushed the uploadPretrainedModelForTraining branch 2 times, most recently from 45d4a17 to 0b094cb Compare August 4, 2016 23:51

lukeyeager reviewed Aug 4, 2016
View reviewed changes

Lucaszw force-pushed the uploadPretrainedModelForTraining branch from 0b094cb to 89bbd5f Compare August 5, 2016 01:38

Upload Pretrained Models for Training

d365d17

changed os.rename to shutil.move

Lucaszw force-pushed the uploadPretrainedModelForTraining branch from 89bbd5f to d365d17 Compare August 5, 2016 19:13

lukeyeager merged commit 4dd28e6 into NVIDIA:master Aug 9, 2016

lukeyeager mentioned this pull request Oct 31, 2016

Moving jobs to another machine #1035

Closed

SlipknotTN pushed a commit to cynnyx/DIGITS that referenced this pull request Mar 30, 2017

Merge pull request NVIDIA#896 from Lucaszw/uploadPretrainedModelForTr…

1bfd493

…aining Upload Pre-trained Models for Fine Tuning.

		@@ -0,0 +1,3 @@
		# Copyright (c) 2014-2016, NVIDIA CORPORATION. All rights reserved.

Upload Pre-trained Models for Fine Tuning. #896

Upload Pre-trained Models for Fine Tuning. #896

Uh oh!

Conversation

Lucaszw commented Jul 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IsaacYangSLA commented Jul 13, 2016

Uh oh!

Lucaszw commented Jul 13, 2016

Uh oh!

jmancewicz Jul 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lucaszw Jul 18, 2016

Choose a reason for hiding this comment

Uh oh!

jmancewicz commented Jul 18, 2016

Uh oh!

Lucaszw commented Jul 18, 2016

Uh oh!

jmancewicz commented Jul 18, 2016

Uh oh!

Lucaszw commented Jul 19, 2016

Uh oh!

lukeyeager commented Jul 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lucaszw commented Jul 27, 2016

Uh oh!

lukeyeager Aug 1, 2016

Choose a reason for hiding this comment

Uh oh!

lukeyeager commented Aug 1, 2016

Uh oh!

lukeyeager commented Aug 1, 2016

Uh oh!

Lucaszw commented Aug 1, 2016

Uh oh!

lukeyeager commented Aug 2, 2016

Uh oh!

Lucaszw commented Aug 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lukeyeager Aug 4, 2016

Choose a reason for hiding this comment

Uh oh!

lukeyeager Aug 5, 2016

Choose a reason for hiding this comment

Uh oh!

Lucaszw commented Aug 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lukeyeager commented Aug 9, 2016

Uh oh!

lukeyeager commented Aug 9, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Lucaszw commented Jul 11, 2016 •

edited

Loading

jmancewicz Jul 18, 2016 •

edited

Loading

lukeyeager commented Jul 26, 2016 •

edited

Loading

Lucaszw commented Aug 2, 2016 •

edited

Loading

Lucaszw commented Aug 5, 2016 •

edited

Loading