Implement idle-timeout shutdown. #1395

jimmc · 2017-05-30T20:21:24Z

No description provided.

yelsayd · 2017-05-30T20:31:53Z

sources/web/datalab/idleTimeout.ts

+
+const idleCheckIntervalSeconds = 5;    // Seconds to wait between idle checks
+
+let idleTimeoutEnabled = true;  // Allow for explicit disable without clearing other values


I'm not sure I understand this comment. maybe clarify a bit?

I changed the comment on this line and the next. Let me know if this is any better.

yelsayd · 2017-05-30T20:33:33Z

sources/web/datalab/idleTimeout.ts

+
+let idleTimeoutEnabled = true;  // Allow for explicit disable without clearing other values
+let idleTimeoutSeconds = 0;   // Shutdown after being idle this long; disabled if 0 or NaN
+let lastUserActivity: number;  // The epoch, in seconds, of the last user activity


also, can we define what user activity means here? or ideally in a comment at the head of the file

I renamed this variable to be more reflective of reality. This module doesn't define what user activity is.

yelsayd · 2017-05-30T20:37:10Z

containers/datalab/content/run.sh

   export VM_PROJECT="${vm_project}"
   export VM_NAME=$(curl -s "${compute_metadata_url}/instance/hostname" -H "Metadata-Flavor: Google" | cut -d '.' -f 1)
   export VM_ZONE=$(curl -s "${compute_metadata_url}/instance/zone" -H "Metadata-Flavor: Google" | sed 's/.*zones\///')
+   export DATALAB_SHUTDOWN_COMMAND="gcloud compute instances stop ${VM_NAME} --project ${VM_PROJECT} --zone ${VM_ZONE}"


FYI, we can get all three values by importing info.

You mean in the javascript code?
Putting the shutdown command here means it gets set when running in a VM but not otherwise.

but what's the value of this? we can already check if we're in a VM.

By using this environment variable, it is easy for me to set a different shutdown command in my startup.sh that shuts down my local notebook server during development. If you prefer, I can remove this line from this script, and instead add code to idleTimeout.ts that calls getVmInfo, checks to see if we are on a VM, and builds this shutdown command if so.

I see, no I think it's fine to stick with the env var if it's more convenient in code.

yelsayd · 2017-05-30T20:39:45Z

sources/web/datalab/idleTimeout.ts

+    }
+  } else {
+    idleTimeoutSeconds = NaN;   // No shutdown command available
+    logging.getLogger().debug('No shutdown command, idle timeout is disabled');


this one should probably be info instead of debug, this is the default log level.

yelsayd · 2017-05-30T20:46:18Z

sources/web/datalab/idleTimeout.ts

+
+// Parse a string like '30s' or '1h 15m 5s' and return seconds.
+// If no input string, or unrecognized stuff, returns NaN.
+function parseInterval(str: string) {


should we grab an npm package for this? something like https://www.npmjs.com/package/parse-duration.

Sounds reasonable. What are the rules regarding adding new packages to what we ship? Licensing guidelines?

MIT license is generally fine.
@Di-Ku just in case.

yelsayd · 2017-05-30T21:23:16Z

sources/web/datalab/static/idle-timeout.js

+  let showDebugLog = false;   // Set to true in the dev console to get log messages during development.
+  function _debugLog(msg) {
+    if (showDebugLog) {
+      console.log(msg);


can we import and use util.debug.log instead?

yelsayd · 2017-05-30T21:30:36Z

sources/web/datalab/static/idle-timeout.js

+        window.setTimeout(_alertIdleShutdown, delayToLetOtherHandlersRun);
+      }
+    }
+    timeoutInfo.enabled = false;  // Make sure we don't get a false indication later.


should this go inside the inner if statement?

No, I want it to be disabled any time this method is called. Changed the method name to be clear about that.

yelsayd · 2017-05-30T21:35:10Z

sources/web/datalab/static/idle-timeout.js

+    _debugLog('== updating timeout display');
+    _debugLog(dropdown);
+    if (timeoutInfo.enabled) {
+      let secondsRemaining = Math.floor((timeoutInfo.expirationTime - Date.now()) / 1000);


I really don't think this should be in seconds, because of the fact that the client code isn't always in sync with the server state. the user will likely see the timer go down second by second, then jump up when the server resets and the client polls the new state.
minutes makes a lot more sense to me.

A few lines below, the call to roundToApproximateTime will generally round to minutes or larger chunks. The user will only see seconds when the time gets down to the last couple of minutes. I find seeing the seconds to be useful for development work. I expect users will rarely see that.

yelsayd · 2017-05-30T21:40:30Z

sources/web/datalab/static/idle-timeout.js

+    _debugLog('== Changing enabled to ' + newValue);
+    updateTimeoutInfo(dropdown);
+    xhr(timeoutUrl, () => {
+      timeoutInfo.enabled = newValue;   // Display the new value right away.


I'd also put the UI update here in the success callback

It is there a couple of lines below. Or did you mean something besides a call to updateTimeoutInfo?

yelsayd · 2017-05-30T21:48:55Z

sources/web/datalab/static/notebook-app.js

    });

+    idleTimeout.setupKernelBusyHeartbeat();
+    $('#mainArea').scroll(idleTimeout.notebookScrolled);


I think we'll need a different div to trigger this event in the non-notebook pages

I haven't played around as much with non-notebook pages. The list page is very reset-happy, it seems to refresh the list of notebooks pretty often, so I didn't bother adding anything else there.

Yea I think you're right about the list page, the sessions page also uses the same code so it'll do the same refresh requests.
We have to think about other pages though, for now we only have the editor and terminal pages, which both sound like the user can spend some time on. Can we make sure we have those covered?

I will take a look at those pages.

yelsayd · 2017-05-30T22:53:17Z

containers/datalab/content/run.sh

   export VM_PROJECT="${vm_project}"
   export VM_NAME=$(curl -s "${compute_metadata_url}/instance/hostname" -H "Metadata-Flavor: Google" | cut -d '.' -f 1)
   export VM_ZONE=$(curl -s "${compute_metadata_url}/instance/zone" -H "Metadata-Flavor: Google" | sed 's/.*zones\///')
+   export DATALAB_SHUTDOWN_COMMAND="gcloud compute instances stop ${VM_NAME} --project ${VM_PROJECT} --zone ${VM_ZONE}"


but what's the value of this? we can already check if we're in a VM.

yelsayd · 2017-05-30T22:58:17Z

sources/web/datalab/idleTimeout.ts

+  return total;
+}
+
+export function resetBasedOnPath(path: string) {


Never mind, I missed this was actually called before handling requests.

yelsayd · 2017-05-30T23:01:36Z

sources/web/datalab/static/notebook-app.js

    });

+    idleTimeout.setupKernelBusyHeartbeat();
+    $('#mainArea').scroll(idleTimeout.notebookScrolled);


Yea I think you're right about the list page, the sessions page also uses the same code so it'll do the same refresh requests.
We have to think about other pages though, for now we only have the editor and terminal pages, which both sound like the user can spend some time on. Can we make sure we have those covered?

yelsayd · 2017-05-30T23:03:30Z

test/unittests/static/idle-timeout_spec.js

+    it('converts single-unit strings correctly', function() {
+      expect(idleTimeout._secondsToString(1)).toEqual('1s');
+      expect(idleTimeout._secondsToString(15)).toEqual('15s');
+      expect(idleTimeout._secondsToString(2*60)).toEqual('2m');


one more test with an unrounded number, e.g. 260+1 or 360*60+61?

Multi-unit outputs are in the next test, a few lines lower. Is there something you want me to add there?

Nope, I think it's all covered.

Implement idle-timeout shutdown.

11afeeb

jimmc requested review from yelsayd, ojarjur and chmeyers May 30, 2017 20:21

yelsayd suggested changes May 30, 2017

View reviewed changes

jimmc added 2 commits May 30, 2017 14:58

Fix one-character typo that caused test failures.

f66d3f2

Respond to PR comments.

2b65a18

yelsayd reviewed May 30, 2017

View reviewed changes

jimmc and others added 2 commits June 1, 2017 11:10

Make idleTimeout user setting work.

fb68aa3

Merge branch 'master' into jimmc/1264-vm-timeout

403f534

yelsayd approved these changes Jun 1, 2017

View reviewed changes

Merge branch 'master' into jimmc/1264-vm-timeout

bd15693

jimmc merged commit 460d186 into master Jun 2, 2017

jimmc deleted the jimmc/1264-vm-timeout branch June 2, 2017 00:37

jimmc mentioned this pull request Jun 2, 2017

Automatic VM Lifetime Management #1264

Closed

chmeyers mentioned this pull request Aug 18, 2017

Release 1.2.20170818 of the datalab command line tool. #1593

Closed


		const idleCheckIntervalSeconds = 5; // Seconds to wait between idle checks

		let idleTimeoutEnabled = true; // Allow for explicit disable without clearing other values

Implement idle-timeout shutdown. #1395

Implement idle-timeout shutdown. #1395

Uh oh!

Conversation

jimmc commented May 30, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!