Scheduler Stage 3 #488

mfix22 · 2018-07-11T23:37:45Z

(Moved PR #487 here)

This is a WIP PR that adds cronInterval support to the frontend 👍

We will shortly add the backend support and merge it in with this branch. We just wanted to get this up for @n-riesco to review before doing so.

TODO:

Add backend support for scheduling
Finish off unit testing components
Run Prettier

Fix linting from Prettier get initial cron mode id from query Add login button to preview modal Move getInitialCronMode to cron-helpers Fix linting from Prettier get initial cron mode id from query Add login button to preview modal Move getInitialCronMode to cron-helpers

n-riesco · 2018-07-12T07:59:40Z

app/components/Settings/scheduler/scheduler.jsx

 import ReactDataGrid from 'react-data-grid';
 import ms from 'ms';
 import matchSorter from 'match-sorter';
+import * as prettyCron from 'prettycron';


let's use something lighter that doesn't depend on later: https://www.npmjs.com/package/cronstrue

n-riesco · 2018-07-12T08:01:03Z

app/components/Settings/scheduler/preview-modal.jsx


            const initialModeId = getInitialCronMode(props.query);
+
+            const canEdit = this.props.currentRequestor && this.props.currentRequestor === props.query.requestor;


n-riesco · 2018-07-12T17:19:46Z

test/app/components/Settings/scheduler/create-modal.test.jsx

+            .simulate('click');
+
+        // onSubmit is asynchronous
+        await wait();


if i remember correctly, our webpack/babel/mocha setup isn't ready to use await.

Hmm. This worked for me though. Should i remove it and use setTimeout directly instead?

Please, ignore my comment. It doesn't apply any longer. We upgraded to Node 8 recently.

n-riesco · 2018-07-12T17:25:01Z

package.json

  },
  "dependencies": {
    "alasql": "^0.4.5",
+    "cronstrue": "^1.31.0",


move to devDependencies

n-riesco · 2018-07-13T16:40:30Z

test/app/components/Settings/scheduler/scheduler.test.jsx

+          });
+      });
+
+      it('should throw GraphQL errors on failure', () => {


n-riesco · 2018-07-13T16:41:43Z

test/app/components/Settings/scheduler/scheduler.test.jsx

+    });
+
+    describe('Creating Queries', () => {
+      it('should handle success cases', () => {


how about should schedule a query only if logged in?

n-riesco · 2018-07-13T16:42:32Z

test/app/components/Settings/scheduler/scheduler.test.jsx

+    });
+
+    describe('Updating Queries', () => {
+      it('should handle success cases', () => {


how about should update a scheduled query only if logged in?

n-riesco · 2018-07-13T16:42:57Z

test/app/components/Settings/scheduler/scheduler.test.jsx

+          });
+      });
+
+      it('should throw GraphQL errors on failure', () => {


n-riesco · 2018-07-13T16:59:34Z

Looks good so far. I'd suggest the following changes:

I've tested if it's possible for the Log In button to navigate directly to the Login page and it doesn't seem to break anything:

--- a/app/components/Settings/Settings.react.js
+++ b/app/components/Settings/Settings.react.js
@@ -25,7 +25,7 @@ class Settings extends Component {
         this.renderSettingsForm = this.renderSettingsForm.bind(this);
         this.updateSelectedPanel = this.updateSelectedPanel.bind(this);
         this.openScheduler = this.openScheduler.bind(this);
-        this.openLogin = this.updateSelectedPanel.bind(this, 3);
+        this.openLogin = () => window.location.assign('/login');
         this.state = {
             editMode: true,
             selectedPanel: {},

(after the on-prem release) I'd like to use a better default than DEFAULT_REFRESH_INTERVAL (I know that the monthly case isn't possible, but we use the maximum value setInterval can take).

How's the work on the backend progressing?

n-riesco · 2018-07-14T08:42:20Z

test/backend/QueryScheduler.spec.js

        const spy = sinon.spy();
-        const refreshInterval = 1; // second
+        const refreshInterval = 5; // 5 seconds
+        const cronInterval = `*/${refreshInterval} * * * * *`;


we should keep the units for testing the old format (i.e. undefined cronInterval).

The problem with keeping refreshInterval is that it is handled differently in the queryScheduler.scheduleQuery call. Currently, refresh intervals are mapped to their corresponding cron expression before being scheduled. If they match one of the supported values, they're converted accordingly, but if they're not recognized as a supported value (as is the case of the value passed in the test), they default to a schedule of every week.

But I can update the code to accept a small enough refreshInterval to test if you'd like? I had assumed you wouldn't want me to alter the desired logic of the scheduler solely to conform to an existing test, which is why I originally opted to update the test units instead.

On second thought, I updated the tests to use fake timers which eliminates this issue. The original refereshInterval format has been restored.

nicolaskruchten · 2018-07-19T20:51:34Z

Cool. Just glancing at the code, there's a WAITING_MESSAGE constant that contains text I've never seen in the UI... when does that show up?

briandennis · 2018-07-19T21:03:41Z

It shows up after taking more than 5 seconds to save/update. If you're trying to test it, we've been adding wait functions to our queries to force longer runtime.

n-riesco · 2018-07-23T14:09:36Z

package.json

    "pg-hstore": "^2.3.2",
    "restify": "^4.3.2",
-    "sequelize": "^3.30.4",
+    "sequelize": "3.30.4",


n-riesco · 2018-07-23T14:10:11Z

package.json

    "bunyan": "^1.8.12",
    "chai": "^4.1.2",
-    "chromedriver": "^2.33.2",
+    "chromedriver": "2.33.2",


Similar to the situation with sequelize, Chromedriver pulled in a minor patch which was throwing on CI (for example here). Freezing the version at 2.33.2 resolved the error.

n-riesco · 2018-07-23T14:26:49Z

@briandennis

about this failure on circleCI: I've recently noticed this failure. It's not a persistent failure. It goes away after a rebuild. I reckon this is an issue with out connector for redshift.
Thank you for fixing the overflow in SQLTable!
I've noticed a small issue while editing queries with a multi-selection of weekdays. See screen capture below:

briandennis · 2018-07-23T15:17:51Z

@n-riesco re the circleCI failure: I tried rebuilding a few times and it continued to break on that specific Redshift test with that error. I then tried connecting to redshift manually and also hit the column "typarray" does not exist in pg_type error while connecting.

After some googling, it looks like this was a backwards compatibility issue with the Postgres version Redshift uses under the hood. The issue was introduced in a minor version of sequelize which is why it got pulled in when updating the yarn.lock in this PR.

Freezing the version at 3.30.4 fixed the problem, but let me know if you think there's a better solution!

briandennis · 2018-07-23T20:34:31Z

@n-riesco the weekday input initialization has been fixed!

I think this PR is at a stage where it's ready to be merged into master. Keep in mind, we're doing at least one more PR after this that addresses any UX updates and bug fixes found while doing final polish and testing before the code freeze. I think we should handle any additional changes there.

What do you think? Are we good to merge this in?

n-riesco · 2018-07-23T21:04:06Z

app/components/Settings/cron-picker/cron-picker.jsx

                </Row>
-                {ModeComponent ? <ModeComponent onChange={this.props.onChange} /> : null}
+                {ModeComponent ? (
+                    // eslint-disable-next-line


if the initialTime is null, we want to avoid overriding the default props for the mode by passing undefined. Setting anything to undefined however causes a lint error.

I'll defer the decision onthis one to @nicolaskruchten .

I'm OK with this, perhaps with a comment above the disable one to explain the reason, as it's not obvious.

n-riesco · 2018-07-23T21:15:16Z

app/components/Settings/cron-picker/cron-helpers.js

 }

+export function mapCronToHourFormat(cronExpression) {
+    if (!cronExpression) {


How about validating the cron expressions using cronstrue (untested)?

try { cronstrue.toString(cronExpression); } catch (_) { return null; }

awesome idea! I added that in

n-riesco · 2018-07-23T21:21:02Z

I'll have another go at testing the last changes tomorrow, but the code looks good to me other than the use of eslint-disable-next-line.

See also the suggestion that cronstrue is used to validate the input to mapCronToHourFormat.

nicolaskruchten · 2018-07-24T00:37:32Z

OK, let's merge this in and have all new changes go into a cleanup commit.

Perhaps in the future when doing multi-stage work like this it would make sense to create a new branch to represent the whole feature and then do small PRs into that branch and then one final review on the branch before it hits master?

nicolaskruchten · 2018-07-24T00:38:31Z

(Could we get the tests to pass first though?)

mfix22 added 7 commits July 10, 2018 15:46

feat(Scheduler): Stage 3

0e3cada

Add cron-picker style improvements

39672a3

Address cron mode estimate comments

d46eb53

Add pretty cron formatter for cron intervals

14d11cb

Only allow updates from current user

ba7e1fd

Default cron to '* * * * *'

842ec07

mfix22 requested a review from n-riesco July 11, 2018 23:37

mfix22 mentioned this pull request Jul 11, 2018

WIP: Scheduler Stage 3 (Cron support) #487

Closed

3 tasks

n-riesco reviewed Jul 12, 2018

View reviewed changes

mfix22 added 2 commits July 12, 2018 09:24

prettycron -> cronstrue

84c606f

Add create-modal unit tests

e281891

n-riesco reviewed Jul 12, 2018

View reviewed changes

Add cronstrue to Preview

a5ff72d

n-riesco reviewed Jul 12, 2018

View reviewed changes

package.json Outdated

},

"dependencies": {

"alasql": "^0.4.5",

"cronstrue": "^1.31.0",

Copy link

Contributor

n-riesco Jul 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to devDependencies

mfix22 added 3 commits July 12, 2018 10:48

Add test for preview/create, fix bugs accordingly

ec42323

Scheduler tests

d8d7bd5

Move cronstrue to devDeps

087562e

n-riesco reviewed Jul 13, 2018

View reviewed changes

mfix22 and others added 5 commits July 13, 2018 10:02

Improve test comments

edd4b94

Update openLogin callback

53b67ec

add cron scheduling support to backend

8182b47

update scheduler tests

fe7685b

Update route tests

7da6474

n-riesco reviewed Jul 14, 2018

View reviewed changes

briandennis changed the title ~~WIP: Scheduler Stage 3 (Cron support)~~ Scheduler Stage 3 Jul 21, 2018

briandennis added 2 commits July 22, 2018 16:54

freeze sequelize version

25d2211

freeze chrome driver

477bd70

n-riesco reviewed Jul 23, 2018

View reviewed changes

package.json

"pg-hstore": "^2.3.2",

"restify": "^4.3.2",

"sequelize": "^3.30.4",

"sequelize": "3.30.4",

Copy link

Contributor

n-riesco Jul 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why?

n-riesco reviewed Jul 23, 2018

View reviewed changes

This was referenced Jul 23, 2018

Hold sequelize at v3.30.4 to workaround incompatibilty with redshift #494

Open

Hold chromedriver at v2.33.2 to workaround CircleCI failure #495

Open

briandennis and others added 10 commits July 23, 2018 13:27

add name prop to backend

0cc33b0

Implement Query name input support

cc94469

query.queryName -> query.name

b07b373

query name input max length = 150

b377031

clean up table + modal responsiveness

27d02bc

only store query name if set

e20d276

(revert) payload.fid -> payload.filename

2ca3b75

search for query 'name's too

122dd60

clean up SQL title styles

39f4323

initialize cron-picker modes based on current cronInterval

d21b156

n-riesco reviewed Jul 23, 2018

View reviewed changes

use cronstrue to validate expression passed to mapCronToHourFormat

6a574e1

n-riesco merged commit 35b029e into master Jul 24, 2018

mfix22 deleted the features/scheduler-stage-3 branch August 8, 2018 01:29


		const initialModeId = getInitialCronMode(props.query);

		const canEdit = this.props.currentRequestor && this.props.currentRequestor === props.query.requestor;

Uh oh!

Scheduler Stage 3 #488

Scheduler Stage 3 #488

Uh oh!

Conversation

mfix22 commented Jul 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n-riesco Jul 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mfix22 Jul 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n-riesco Jul 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n-riesco commented Jul 13, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicolaskruchten commented Jul 19, 2018

Uh oh!

briandennis commented Jul 19, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n-riesco commented Jul 23, 2018

Uh oh!

briandennis commented Jul 23, 2018

Uh oh!

briandennis commented Jul 23, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n-riesco commented Jul 23, 2018

Uh oh!

nicolaskruchten commented Jul 24, 2018

Uh oh!

nicolaskruchten commented Jul 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

mfix22 commented Jul 11, 2018 •

edited

Loading

n-riesco Jul 12, 2018 •

edited

Loading

mfix22 Jul 12, 2018 •

edited

Loading

n-riesco Jul 12, 2018 •

edited

Loading