Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

atombrella
Copy link
Contributor

@atombrella atombrella commented Jul 6, 2018

  • Sort out unneeded casts in lookups, because they unnecessarily cause problems on PostgreSQL.
  • Fix the problem with opclasses and IndexError when a lookup is used.
  • More tests. Suggestions are welcomed.

Copy link
Member

@charettes charettes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice stuff, I've been wanting this feature for a while!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move this 2 lines above, right next to the other change about Indexes.

@atombrella atombrella force-pushed the index_partial branch 3 times, most recently from 55b13c4 to f461550 Compare July 23, 2018 10:24
@MarkusH
Copy link
Member

MarkusH commented Jul 23, 2018

@atombrella what do you need / how can I help to get this over the finish line?

@atombrella
Copy link
Contributor Author

@MarkusH The biggest hurdle now is the unnecessary lookup casts. I posted a question on IRC about it, but perhaps the mailing list is a better place. On PostgreSQL, cast is a mutable function, and table.date_column >= '2007-10-01'::timestamp makes it break. I think I figured out the second thing in the list by adding params=None somewhere in AddIndex to the execute-call.

@atombrella atombrella force-pushed the index_partial branch 2 times, most recently from 8755635 to a6816b2 Compare July 24, 2018 11:48
@atombrella
Copy link
Contributor Author

@felixxm Could you please assist with the Oracle test failures?

@django django deleted a comment from MarkusH Jul 25, 2018
@felixxm
Copy link
Member

felixxm commented Jul 25, 2018

@atombrella Sure, I will check them in this week.

@felixxm
Copy link
Member

felixxm commented Jul 27, 2018

@atombrella Thanks for this PR 🚀 On Oracle we should be able to emulate (in most cases) partial indexes by function-based indexes. I will check how it works with this PR.

@atombrella
Copy link
Contributor Author

@felixxm Aren't function-based indexes what was attempted in #8056 ? I have a local branch trying to rework some of it, but I am stuck how to deal with the expression and columns in the DDL-classes that were introduced after that PR was made.

@felixxm
Copy link
Member

felixxm commented Jul 30, 2018

@atombrella Oracle doesn't index NULL values, therefore all indexes are partial indexes (it's not a 🐛 it's a feature 😄). Example SQL for the PartialIndexTests.test_partial_index:

CREATE INDEX "RECENT_ARTICLE_IDX" ON "INDEXES_ARTICLE" (
    CASE WHEN "PUB_DATE" > '2015-01-01' THEN 1 END
);

@atombrella atombrella force-pushed the index_partial branch 3 times, most recently from cf0dcde to 9a72d67 Compare September 4, 2018 18:48
@GeorgeLubaretsi
Copy link

Awesome stuff!

Is handling of unique indexes planned for this feature? They can be really useful as a conditional alternative for unique constraints.

@charettes
Copy link
Member

@georgebrock there's work being done in this regards in #10337.

Not sure about the API yet but I think the idea would be to have UniqueConstraint accept a condition=Q object as well and have it use a partial unique index under the hood #10271 (comment)

@atombrella
Copy link
Contributor Author

@felixxm Would you be ok with a note about Oracle? I don't think dealing with functional indexes is in the scope of this PR. Besides, since the predicate is an addition to the columns, I'm not sure how a potential rewrite would look like.

Likewise for unique-constraints; this seems to be handled currently in the ongoing UniqueConstraint PR. I think unique as an argument to Index was suggested in index-related PR discussions.

@charettes
Copy link
Member

Likewise for unique-constraints; this seems to be handled currently in the ongoing UniqueConstraint PR. I think unique as an argument to Index was suggested in index-related PR discussions.

yeah we really need to gather consensus on the unique kwarg. Having both UniqueConstraint and Index(unique=True) feels odd to me. I'd prefer if only one of the two existed and I tend to agree with @Ian-Foote in #10271 (comment).

@LilyAcorn
Copy link
Contributor

I'm happy to look into supporting partial unique constraints, building on top of work in this PR and #10337 (based on #10271), but I'd like to get the outstanding work merged first. Do we have a ticket for this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the tzinfo bit for the test? I'm worried relying on get_current_timezone could make the test flaky.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason for the tzinfo bit was the type casting which makes the function mutable, and PostgreSQL raises an error. USE_TZ is False by default; without tzinfo, the test fails with django.db.utils.ProgrammingError: functions in index predicate must be marked IMMUTABLE. I believed I touched this problem about timezones in the documentation, but it may need improvement or clarification.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe include a comment here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes 👍 I'll also try to write a sentence or two about DateField under the postgresql note. I noticed that comparison between dates (and not date+time) won't be converted to a mutable function.

@atombrella atombrella force-pushed the index_partial branch 7 times, most recently from 5a1684a to 9921419 Compare September 13, 2018 07:34
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why I added null=True, so it should probably be removed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about auto_now=True instead of null=True?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests pass without any change to that line, so I'll just revert my addition, and don't plan on changing it again.

@timgraham timgraham changed the title Fixed #29547 -- [WIP] Added support for partial indexes. Fixed #29547 -- Added support for partial indexes. Sep 14, 2018
@charettes
Copy link
Member

After thinking about it a bit more I think there's room for both UniqueConstraint and Index(unique=True). With features such as PostgreSQL's 11 unique index INCLUDE and other tweaks unique indexes might require I feel like there should be a way to define both.

Maybe UniqueConstraint(index_name='foo') could be added later on or we could make Index(unique) register a unique constraint automatically? I'm not sure what's the best API here but having a way to reference unique constraints locally could definitely be useful with regards to API's such as bulk_create that could be augmented to express unique constraint conflict resolution.

e.g.

Appointment.objects.bulk_create(objs, conflict_updates=[
    ConflictUpdate(
        'location_timeslot',  # UniqueConstraint(name)
        Q(prefered_timeslot=False),
        timeslot=Subquery(
            Availability.objects.filter(
                location=OuterRef('location')
            ).values('timeslot').order_by('timeslot')[1]
        ),
    ],
})
ON CONFLICT UPDATE
    SET timeslot = (SELECT timeslot FROM ...) WHERE excluded.prefered_timeslot = false

or warn about get_or_create/update_or_create usage against a set of fields that aren't included in a unique constraint.

Copy link
Contributor

@LilyAcorn LilyAcorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me overall.

Thanks to Ian Foote, Mariusz Felisiak, Simon Charettes, and
Markus Holtermann for comments and feedback.
@timgraham timgraham merged commit a906c98 into django:master Oct 30, 2018
@atombrella atombrella deleted the index_partial branch October 30, 2018 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants