Fixed #11964 -- Added support for check constraints in model Meta. #7615

LilyAcorn · 2016-11-24T18:24:58Z

https://code.djangoproject.com/ticket/11964

LilyAcorn · 2016-11-24T18:27:26Z

django/db/models/constraints.py

This won't work because I don't have access to a suitable compiler. I will also need to add support for as_sql to Q objects.

From #8056, it looks like I can do:

compiler = connection.ops.compiler('SQLCompiler')(None, connection, 'default')

LilyAcorn · 2017-02-19T23:49:41Z

django/db/models/constraints.py

This gets me further. Unfortunately, I appear to need a more complete query instead of None, so Col.as_sql can be evaluated.

LilyAcorn · 2017-02-19T23:51:35Z

tests/migrations/test_operations.py

Long term, I want to be able to use Q(pink__gt=2) here, but for now using this avoids the need to implement Q.as_sql.

LilyAcorn · 2017-02-19T23:52:29Z

tests/migrations/test_state.py

I haven't really thought whether this is the best way to fix these tests, but it is sufficient for now.

LilyAcorn · 2017-02-26T17:35:50Z

I'm not sure where is best to put MySQL error handling. It supports the syntax, but doesn't actually do anything with it, so I think we should fail loud somewhere. Perhaps in a check?

LilyAcorn · 2017-02-26T17:45:01Z

I'm also not sure how best to handle SQLite. It appears to support the feature, but I get a syntax error from SQLite with the syntax that works on postgres. I think this is because that uses ALTER TABLE, which SQLite only supports a limited form of, but I'm not clear what to do instead.

Django does set supports_column_check_constraints to False for SQLite's db features, so perhaps working around this limitation would be difficult.

LilyAcorn · 2017-02-26T21:20:52Z

@jarshwah @MarkusH I'd appreciate your feedback on this. There's definitely some work left to do - docs, MySQL, SQLite, naming consistency, any edge-cases I've missed - but I think the core is mostly there.

I'd also like to add a test to check an IntegrityError is actually raised if a Check constraint is violated, but I'm not sure how to set up the model to test so the Check is applied in the database.

auvipy · 2017-03-18T15:26:15Z

docs/ref/models/options.txt

should be 2.0

LilyAcorn · 2017-04-17T15:16:34Z

This now just needs to do something sensible for MySQL and the rest is cleanup and edge-cases.

LilyAcorn · 2017-04-17T15:17:48Z

django/db/models/sql/query.py

It may be possible to simplify this further. I'm also not 100% certain I've thought of every edge-case, so I may have accidentally deleted something important.

It feels odd why so much code needs to be duplicated. Have you considered letting Query subclass this?

I hadn't considered that. My main focus has been trying to get something that works well for this use case. Refactoring to allow code re-use is a secondary concern.

This is a lot of stuff to duplicate, especially as you note below that you don't understand everything. I'm concerned that future expression refactors/features will be missed in constraints and things will asplode.

Yeah - that's a good point. I'll take another pass and see if I can minimise the amount of duplicated code.

I'm out of my depth here; my instinct would be to figure out what the responsibilities of Q / Query / etc. are, which bits are needed for constraints, and refactor to share these bits.

Malcolm's "Dungeon Master's guide to Django's ORM" talk may help.

For example, _force_pk (duplicated below) was removed in ec50937.

LilyAcorn · 2017-04-17T15:18:28Z

django/db/models/sql/query.py

I've not put any significant effort into understanding this method yet, so it may well be possible to simplify.

LilyAcorn · 2017-04-17T15:20:01Z

tests/queries/test_query.py

Ideas for more tests to add here would be appreciated.

jackton1 · 2018-04-30T20:48:30Z

django/db/models/sql/query.py

kwargs = {'reuse': can_reuse, 'allow_joins': allow_joins} if isinstance(value, F): kwargs['simple_col'] = simple_col value = value.resolve_expression(self, **kwargs)

carltongibson · 2018-05-17T09:08:36Z

Hi @Ian-Foote. Thanks for the extended effort here.

Reviewing the history there are just a few small points outstanding:

We'll have to target 2.2 now, so those references need updating.
Via @aaugustin: "This needs a changelog entry."
Via @timgraham: "The migration operations should be documented in docs/ref/migration-operations.txt."

I pulled in your test for the IntegrityError here. That already behaves as expected.

I was going to say, can we make the MySQL add_constraint()/remove_constraint() no-ops, with an extra warning maybe? But the test failed without crashing when run on MySQL. I then noticed your comment:

[MySQL] supports the syntax, but doesn't actually do anything with it...

Thus I think we're good there.

I would push the test straight here but I didn't understand this:

...I'm not sure how to set up the model to test so the Check is applied in the database.

Perhaps that's not relevant any more?

Other than that, this looks super!

LilyAcorn · 2018-05-17T16:30:30Z

Oh that's annoying - I didn't notice the 2.1 cut-off approaching or I'd have flagged this a bit earlier.

...I'm not sure how to set up the model to test so the Check is applied in the database.

I don't remember where I said that, so I'm guessing it's not relevant any more.

I'll make sure the outstanding points about documentation are resolved at the DjangoCon Europe sprints.

felixxm · 2018-05-17T16:55:04Z

django/db/backends/base/schema.py

I think we can use unpacking generalization instead of chain, i.e.

sql = self.sql_create_table % { 'table': self.quote_name(model._meta.db_table), 'definition': ', '.join((*column_sqls, *constraints)), }

We probably can, but I don't think it's better.

but it's faster because there's no function call overhead

In [1]: from itertools import chain In [2]: a = ['1', '2'] In [3]: b = ['buckle', 'my', 'shoe'] In [4]: %timeit list(chain(a, b)) 506 ns ± 2.19 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) In [5]: %timeit list((*a, *b)) 313 ns ± 4.46 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Ok - I'll do it then.

felixxm · 2018-05-17T19:23:10Z

django/db/models/constraints.py

path is unused, so we can use a _ instead.

felixxm · 2018-05-17T19:24:24Z

django/db/models/constraints.py

Please use hanging indentation.

I'm not quite clear what style you're recommending here - can you be more explicit?

hanging = ( indentation, has, a, newline, after, opening, bracket, )

felixxm · 2018-05-17T19:30:05Z

django/db/models/constraints.py

quote_name is a temporary variable that's used only once, therefore I think we can remove it and use schema_editor.quote_name directly, i.e.:

'name': schema_editor.quote_name(self.name),

felixxm · 2018-05-17T19:38:16Z

django/db/models/constraints.py

using argument is unused and IMO can be removed (the same in create_sql()).

felixxm · 2018-05-17T19:42:36Z

django/db/migrations/operations/models.py

This can be single-lined.

LilyAcorn · 2018-05-26T09:20:11Z

@carltongibson I think I've resolved the outstanding points now. I've included the test you linked, added the missing docs and fixed @felixxm's comments.

carltongibson · 2018-05-29T09:30:22Z

OK. Great stuff @Ian-Foote! I will have a look again this week.

carltongibson

Hi @Ian-Foote.

Thanks for making the changes.

I have a few questions which might mean we need to break it up a little.

It's seems there's three parts to it:

The new tests on Query in tests/queries/test_query.py.
The introduction of SimpleCol
Adding the CheckConstraint (and related).

So the tests on Query seem fine but largely unrelated. They reference SimpleCol but could/should they not be introduced on their own? (It's the final file in the diff, so I got there and was like What's this doing here?)

Then SimpleCol exists to introduce different SQL to Col right?

Is it worth a docstring explaining why it exists?
Do we need some tests around this? (Neither Col nor Ref make much appearance in the test suite...) Maybe around the _get_col() helper and the changed (?) behaviour when simple_col is True?

Finally the constraints stuff builds on this, and is great.

I guess my observation if just that, whilst the constraints changes are totally clear, the SimpleCol addition (and Query tests) are slightly unexplained. Why are they all in one commit, rather than 2 or 3, with maybe a few extra test cases, particularly on SimpleCol?

Perhaps it's fine as is, but could you comment on that? 🙂

Ta. (Good stuff!)

MarkusH

This looks great from where I'm standing. Thanks @Ian-Foote! We've talked about this quite a bit so I know the answers to most of the questions @carltongibson asked. But it doesn't hurt having them explained here / in the code.

I'm looking forward to this!

carltongibson · 2018-06-06T14:19:54Z

Yes. Just to emphasise, I totally agree: this is great. My queries are merely clarificatory.

LilyAcorn · 2018-06-06T16:13:52Z

Thanks both!

So the tests on Query seem fine but largely unrelated. They reference SimpleCol but could/should they not be introduced on their own?

These are a remnant from when I tried to introduce a simplified Query class for just the check constraints functionality. I kept them when I abandoned that approach because they worked to test my changes to Query itself.

Then SimpleCol exists to introduce different SQL to Col right?

Yes - specifically, Col produces SQL including an alias that is syntactically invalid in a check constraint, so we use SimpleCol instead to omit this.

Do we need some tests around this? Maybe around the _get_col() helper and the changed (?) behaviour when simple_col is True?

I think this is mostly covered by the additional tests on Query and the tests of check constraints themselves. simple_col is currently only set to True by my check constraints work.

If you think any of this explanation would work well in a docstring somewhere, let me know and I'll add it. I'm also happy to add some more tests around SimpleCol if you do think it's worth it.

carltongibson · 2018-06-06T16:46:51Z

Hey @Ian-Foote.

Thanks for that. Super. I think, for me, if you could add a docstring to SimpleCol that would be handy.

Beyond that, lets call it RFC. Good work.

(Now can we just get forms to parse constraints for validation there...? 🙂)

LilyAcorn · 2018-06-12T17:50:49Z

@carltongibson Docstring added.

adamchainz · 2018-07-11T10:37:22Z

🎉 Thanks for your great work and persistence, @Ian-Foote and everyone else 🎉

LilyAcorn commented Nov 24, 2016

View reviewed changes

LilyAcorn force-pushed the ticket_11964 branch 2 times, most recently from f7f256c to 89b2092 Compare February 19, 2017 23:46

LilyAcorn commented Feb 19, 2017

View reviewed changes

LilyAcorn force-pushed the ticket_11964 branch 4 times, most recently from fc2f707 to 835f04e Compare February 26, 2017 16:40

LilyAcorn force-pushed the ticket_11964 branch from 835f04e to d928c31 Compare February 26, 2017 21:03

LilyAcorn force-pushed the ticket_11964 branch 4 times, most recently from 50f4ae0 to 5e97e81 Compare February 27, 2017 20:56

auvipy reviewed Mar 18, 2017

View reviewed changes

docs/ref/models/options.txt Outdated

Copy link

Contributor

auvipy Mar 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be 2.0

LilyAcorn force-pushed the ticket_11964 branch 3 times, most recently from 00ee601 to 505beef Compare April 8, 2017 09:43

LilyAcorn mentioned this pull request Apr 10, 2017

Fixed #28043 -- Stopped AddIndex.state_forwards mutating shared state #8315

Merged

LilyAcorn force-pushed the ticket_11964 branch 2 times, most recently from 44e6a30 to 76217f8 Compare April 17, 2017 15:14

LilyAcorn commented Apr 17, 2017

View reviewed changes

charettes mentioned this pull request Jun 10, 2017

[WIP] Fixed #20581 -- Added class based index Unique and UniqueDeferred. #8620

Closed

LilyAcorn force-pushed the ticket_11964 branch 2 times, most recently from 87f416f to f416afe Compare August 5, 2017 22:22

LilyAcorn force-pushed the ticket_11964 branch from 0b79d6b to 72bda5d Compare April 3, 2018 21:12

jackton1 reviewed Apr 30, 2018

View reviewed changes

LilyAcorn force-pushed the ticket_11964 branch from 72bda5d to 6b60301 Compare May 6, 2018 18:21

felixxm reviewed May 17, 2018

View reviewed changes

LilyAcorn force-pushed the ticket_11964 branch from 6b60301 to 42c8d17 Compare May 26, 2018 09:18

LilyAcorn force-pushed the ticket_11964 branch 2 times, most recently from 65021bb to 029c0b4 Compare May 26, 2018 09:27

carltongibson self-requested a review May 29, 2018 09:30

carltongibson reviewed May 30, 2018

View reviewed changes

MarkusH approved these changes Jun 6, 2018

View reviewed changes

carltongibson approved these changes Jun 6, 2018

View reviewed changes

LilyAcorn force-pushed the ticket_11964 branch from 029c0b4 to 1a5ade0 Compare June 12, 2018 17:50

timgraham force-pushed the ticket_11964 branch 2 times, most recently from 3429e63 to c979ab4 Compare July 10, 2018 19:19

Fixed django#11964 -- Added support for database check constraints.

952f05a

timgraham force-pushed the ticket_11964 branch from c979ab4 to 952f05a Compare July 10, 2018 19:44

timgraham merged commit 952f05a into django:master Jul 10, 2018

LilyAcorn deleted the ticket_11964 branch July 10, 2018 21:42

charettes mentioned this pull request Aug 6, 2018

Fixed #29641 -- Added UniqueConstraint support. #10271

Closed

schinckel mentioned this pull request Dec 1, 2018

Preparation for Filter Expressions/Expression Filters. #10098

Closed

charettes mentioned this pull request Dec 27, 2018

Fixed #30062 -- Added support for unique conditional constraints. #10796

Merged

Uh oh!

Fixed #11964 -- Added support for check constraints in model Meta. #7615

Fixed #11964 -- Added support for check constraints in model Meta. #7615

Uh oh!

Conversation

LilyAcorn commented Nov 24, 2016 • edited by timgraham Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LilyAcorn commented Feb 26, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LilyAcorn commented Feb 26, 2017

Uh oh!

LilyAcorn commented Feb 26, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LilyAcorn commented Apr 17, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carltongibson commented May 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LilyAcorn commented May 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LilyAcorn commented Nov 24, 2016 •

edited by timgraham

Loading

LilyAcorn commented Feb 26, 2017 •

edited

Loading

LilyAcorn commented Feb 26, 2017 •

edited

Loading

carltongibson commented May 17, 2018 •

edited

Loading

LilyAcorn commented May 17, 2018 •

edited

Loading

carltongibson left a comment •

edited

Loading

carltongibson commented Jun 6, 2018 •

edited

Loading