Multivariate Linear Regression Example#108
Conversation
|
Hi @caisq, This looks ready for first review. |
bileschi
left a comment
There was a problem hiding this comment.
Hi Manraj, thanks for putting this together! it looks like this is designed for node. Is it possible to adjust this to instead focus on the browser? A pattern for loading data without a file system can be cloned from the mnist example. After this, a simple index.html and ui.js should do the trick. Thanks!
Reviewable status: 0 of 1 LGTMs obtained
multivariate-linear-regression/data.js, line 3 at r1 (raw file):
Copyright 2018 Google LLC. All Rights Reserved.
This should read
Copyright 2018 the tfjs-examples Authors.
multivariate-linear-regression/data.js, line 130 at r1 (raw file):
this.dataset[2] = normalizeDataset(this.dataset[2]);
Technically, the normalization parameters are learned parameters. They should be estimated from the training set and applied to the test set.
multivariate-linear-regression/index.js, line 21 at r1 (raw file):
const timer = require('node-simple-timer');
would Date().getTime(); work for these?
nsthorat
left a comment
There was a problem hiding this comment.
I think it's fine to keep it node since we're lacking node tutorials -- however can you name the directory as such: "multivariate-linear-regression-node"?
Nice work Manraj, thank you!
Reviewable status: 0 of 1 LGTMs obtained
multivariate-linear-regression/data.js, line 3 at r1 (raw file):
Previously, bileschi (Stanley Bileschi) wrote…
Copyright 2018 Google LLC. All Rights Reserved.This should read
Copyright 2018 the tfjs-examples Authors.
I think it's fine to keep as is -- none of the other examples do this.
multivariate-linear-regression/data.js, line 33 at r1 (raw file):
// Downloads a test file only once and returns the csv async function loadCsv(filename) {
put this in and readCsv in utils (since their API is pretty easy to understand)
multivariate-linear-regression/data.js, line 70 at r1 (raw file):
// Shuffles data and label using Fisher-Yates algorithm. const shuffle = (data, label) => {
put this in utils
multivariate-linear-regression/data.js, line 130 at r1 (raw file):
Previously, bileschi (Stanley Bileschi) wrote…
this.dataset[2] = normalizeDataset(this.dataset[2]);Technically, the normalization parameters are learned parameters. They should be estimated from the training set and applied to the test set.
+1 -- they don't necessarily have to be learned but the normalization parameters should span both training and test set (computing mean / variance across the entire dataset).
multivariate-linear-regression/data.js, line 174 at r1 (raw file):
} _generateBatch(isTrainingData, batchSize) {
no need for undescores
multivariate-linear-regression/data.js, line 223 at r1 (raw file):
} module.exports = new BostonHousingDataset();
use export const BostonHousingDataset
multivariate-linear-regression/index.js, line 28 at r1 (raw file):
const TEST_SIZE = 173; const LEARNING_RATE = 0.01;
any reason not to use the layers API? will make the model definition much simpler
multivariate-linear-regression/utils.js, line 1 at r1 (raw file):
// Calculate the arithmetic mean of a vector.
license at the top of thise file
multivariate-linear-regression/utils.js, line 37 at r1 (raw file):
}; module.exports = {
you cant just export const stddev? why module.exports?
|
Hi Stanley,
Would this example be completely browser focused?
I'll have a look but current dataset contains
Agreed, will fix this
For web, we have |
nsthorat
left a comment
There was a problem hiding this comment.
Ignore me regarding Node vs Browser, didn't see the other conversation. Browser SGTM.
Reviewable status: 0 of 1 LGTMs obtained
|
Made the changes Apologies, I'm not comfortable with permissions asked by Reviewable and hence don't make use of it.
Done Made use of
There is already an example for regression making use of Layers API.
Okay
I feel we should compute mean and variance on the training set and apply it to testing to avoid any leakage of information. I've made the changes accordingly. Please correct me if I'm wrong.
Done
Done
Done
Done
Done
I'm not aware of nodejs version for which examples should be compatible and hence went with it |
caisq
left a comment
There was a problem hiding this comment.
Thanks, Manraj!! In this first round of review, I made some high-level comments. I will dive deeper once you have addressed these comments.
Reviewable status: 0 of 1 LGTMs obtained
multivariate-linear-regression-core/data.js, line 66 at r3 (raw file):
utils.shuffle
This custom shuffling logic won't be necessary if you use the Layers API's fit() method, which performs shuffling by default. (See my comment below.)
multivariate-linear-regression-core/utils.js, line 34 at r3 (raw file):
/**
I suggest we write full JSDocs with arguments and return values included, see example at:
https://github.com/tensorflow/tfjs-examples/blob/master/lstm-text-generation/index.js#L48
multivariate-linear-regression-core/utils.js, line 35 at r3 (raw file):
nit: The comment should be separated from the asterisk by one space.
To avoid the overhead of manual formatting, you can run
clang-format -i --style=google util.js
and also on other js files.
multivariate-linear-regression-core/utils.js, line 35 at r3 (raw file):
csv
Nit: end comment lines with period per Google style.
multivariate-linear-regression-core/utils.js, line 55 at r3 (raw file):
export const shuffle
Is there any reason why this is an arrow function, while others like loadCsv are normal functions? If not, can we make then consistent?
multivariate-linear-regression-core/README.md, line 3 at r3 (raw file):
Multivariate Linear Regression
For beginners, please briefly explain what Multivariate Linear Regression is, something like, "linear regression with more than one numerical input feature".
multivariate-linear-regression-core/README.md, line 3 at r3 (raw file):
Boston Housing Dataset.
For beginners, please briefly explain the background of this dataset, possibly by referring to https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html
multivariate-linear-regression/index.js, line 28 at r1 (raw file):
Previously, nsthorat (Nikhil Thorat) wrote…
any reason not to use the layers API? will make the model definition much simpler
+1.
We plan to start the teaching material from the simpler layers API. Can you please rewrite this in that API. It'll take only a single tf.layers.dense layer and it'll also simply the training code.
…ntation, inconsistencies in syntax
|
@caisq Made the changes. Any inputs on the user interface? I was thinking of adding plot for train and test loss. |
caisq
left a comment
There was a problem hiding this comment.
Reviewable status: 0 of 1 LGTMs obtained
multivariate-linear-regression-core/index.js, line 34 at r4 (raw file):
kernelInitializer: 'randomNormal',
Just curious: Why use this instead of the default glorotNormal? Same for the line below.
multivariate-linear-regression-core/index.js, line 36 at r4 (raw file):
useBias: true
This is true by default. So no need for this line.
multivariate-linear-regression-core/index.js, line 46 at r4 (raw file):
const history = await model.fit( batch.data, batch.target, {batchSize: BATCH_SIZE, shuffle: true});
It is my vote that we feed the whole dataset as a single xs tensor and a single ys tensor to this fit() call. The fit() method will take care of the batching and shuffling by itself, under the hood. This will greatly simply data.js.
Remember this is the 2nd example we will show in the teaching material. We still want to keep it simple.
Also, please note that shuffle is true by default. So no need to specify it explicitly here.
multivariate-linear-regression-core/index.js, line 50 at r4 (raw file):
Quoted 5 lines of code…
if (step && step % 2 === 0) { const loss = history.history.loss[0].toFixed(6); console.log(` - step: ${step}: loss: ${loss}`); }
This should be replaced with an onEpochEnd callback option to fit(). See example at: https://github.com/tensorflow/tfjs-examples/blob/master/iris/index.js#L64
multivariate-linear-regression-core/index.js, line 53 at r4 (raw file):
await tf.nextFrame();
This should go into the onEpochEnd callback. Please also add a comment here about what this is for.
multivariate-linear-regression-core/index.js, line 66 at r4 (raw file):
model.predict(evalData.data);
This entire function can and should be replaced with a single call to model.evaluate(). See example at:
caisq
left a comment
There was a problem hiding this comment.
Thanks a lot for making the changes for far, @manrajgrover. It's getting very close! I just have a small number of remaining comments.
Reviewable status: 0 of 1 LGTMs obtained
multivariate-linear-regression-core/index.js, line 34 at r5 (raw file):
.fit(trainData.data, trainData.target, {
We need a validationSplit for this fit(). Validation loss is important because it tells us when to stop the training, so that the value of NUM_EPOCHS can be less arbitrary.
Maybe set validationSplit to a value around 0.15.
Also, if the original training data CSV is not in a randomized order, using validationSplitwill require shuffling the data, because the validationSet is always the fraction of the data at the end of xs and ys. I may have previously advised you to remove random shuffling in data.js - if that's the case, sorry about the back and forth. But here we just need to shuffle once, i.e., not for every epoch.
multivariate-linear-regression-core/index.js, line 36 at r5 (raw file):
1
Can we set this to NUM_EPOCHS, so that we can get rid of the loop in the run function below?
multivariate-linear-regression-core/index.js, line 65 at r5 (raw file):
time
IMO, benchmarking the training speed is not necessary for this example. We aim for simplicity and essence in this example. Can we remove the benchmarking code?
caisq
left a comment
There was a problem hiding this comment.
Reviewable status: 0 of 1 LGTMs obtained
multivariate-linear-regression-core/index.js, line 34 at r6 (raw file):
num_features
nit: Use camelCase for consistency. Same elsewhere in this PR.
multivariate-linear-regression-core/index.js, line 51 at r6 (raw file):
Test
--> Validation. We will use "validation" and "test" to mean different things. "Validation" is the held-out data that we run evaluation on during training. It is used for things like deciding when to stop training. "Test" is the held-out data that the model never sees during training. It is used only after the training completes.
caisq
left a comment
There was a problem hiding this comment.
Reviewable status: 0 of 1 LGTMs obtained
multivariate-linear-regression-core/utils.js, line 22 at r6 (raw file):
'https://gist.githubusercontent.com/ManrajGrover/a4b2b6bf0abda231b4b49af8b9950688/raw/661367f1ab938642ff0d216276b77ace5d288b04/';
Please use this version of Boston Housing dataset instead: https://www.datasciencecentral.com/profiles/blogs/boston-housing-dataset-without-the-racial-profiling-field
Thanks.
@caisq In this case, although model is being used to predict on test data after every epoch, early stopping is not being decided on it. It is being decided based on validation data which I assume
I'll move it out of the callback. |
|
@manrajgrover Not sure I fully understood your last comment. But to answer your question: yes, model.fit() returns loss values for both the training and validation sets if validatoinSplit is set to a value >0. For example, I actually think getting the validation loss from the return value of model.fit() is better than getting it from the callbacks, because of simpler code. This is the 2nd concrete code example of the teaching material and we are probably not quite ready to dive into the details of callbacks yet. In any case, I think plotting val_loss alongside loss is important because it shows the important concept of underfitting and overfitting. Running |
@caisq Thanks for sharing this. Using callback will enable plotting a live graph. We can surely make use of Regarding early stopping, I'm not sure if
Agreed |
|
@manrajgrover Yep. Sorry what I wrote before might be misleading. Early stopping is not done in your code. To do early stopping, you need a callback like https://keras.io/callbacks/#earlystopping, which is not implemented in tensorflow.js yet. For this example, showing plots of training and validation losses (along with a test loss value) suffices. There is no need to actually do early stopping. |
|
Looking forward to your next revision before I can approve this PR. Thanks. |
…over/tfjs-examples into multi-linear-regression
|
@caisq Done |
caisq
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 1 of 1 LGTMs obtained
caisq
left a comment
There was a problem hiding this comment.
Thanks, @manrajgrover This looks great!
|
@caisq Thanks for reviewing. If you could put these CSV's on Google Cloud Storage and share the url, I can update the link and if required, processing part accordingly. |
|
@manrajgrover Let's do that later. We will need to settle on a centralized, uniform scheme for storing those data files. When we are done with that, we will update the URLs like this one. Thanks. |
|
@bileschi @ericdnielsen let us know if you have any remaining comments. |
The review has been ongoing for about two weeks. Manraj has addressed multiple rounds of comments. I am going to merge the PR now so we can iterate on it.
This PR adds multivariate linear regression example.
This change is