Hi, thanks for the PR. Please do read the issue charmander linked for you, namely my closing comment:

#2 (comment)

It is not correct to simply discard the indexes returned by Postgres by default. We are choosing to map the starting index from PG (1) to JS's equivalent (0). This seems to bite people who slice arrays via [0:.

The data/query is wrong and you want the parser to fix it. That's fine, but IMO we can't turn things like that on by default that discard information when changing the result into a JS type.

I'd be happy to consider index-stripping as an option which you'd enable yourself when calling setTypeParser. For most users, their intervention should just be to fix their queries. Having an option would at least give us a place to document the underlying problem so users have the option to fix it if convenient.

I think you are not right in several points:

It is not correct to simply discard the indexes returned by Postgres by default. We are choosing to map the starting index from PG (1) to JS's equivalent (0). This seems to bite people who slice arrays via [0:.

I pointed to pgjdbc how they handle it. I trust this code because it is running in production for 6 years, and survived PG upgrades from 9.1 to 10.4.
That is why i replicated their code, just skipping indexes.

The data/query is wrong and you want the parser to fix it. That's fine, but IMO we can't turn things like that on by default that discard information when changing the result into a JS type.

My query is just select a.social_profile_ids from attendee a where social_profile_ids is _int8 column. It is not query problem it is driver problem.

I'd be happy to consider index-stripping as an option which you'd enable yourself when calling setTypeParser. For most users, their intervention should just be to fix their queries. Having an option would at least give us a place to document the underlying problem so users have the option to fix it if convenient.

Ok, but i still think pgjdbc logic more correct. I can add option if you think it is right. But i still think it should be turned on by default, because as i said it is not query problem.

Returning to pgjdbc point. The code added to that driver skipping indexes was added 14 years ago.

brianc/node-pg-types#75 (comment)

jbdc admits to fixing it for you because that's the way it was, it's kinda circular. I don't understand how you're getting indexes in queries given an _int8 column in the first place.

Maybe storage respects indexes if set, I don't know. If you can produce a test script that shows that 0-indexes come back without setting them at some point, that's enough weight towards stripping by default.

I'd also be amenable to more significant research. You used jbdc, but show me where Rails, Django, database/sql, or a few other popular clients' parsers discard the indexes. I'd be interested in trying to dig through the history and see their discussions.

If everyone else has given up on this issue, I will too. But "pgjbdc did it forever ago" isn't enough for me. Especially b/c pgjbdc did it to preserve backwards compat when graduating from pg@8 to 9. I'd prefer to be technically correct (i.e. preserve indexes) but since that's impossible I defer to community consensus. But not to java 🙂.

That is fair point. I will take a look what is happenning in other drivers and ask in pg mailing list what they think about it.
Thank you for very reasonable and supporting point of view.

I do not have any special setup for this column. It was just added to old table with

alter table attendee
	add column if not exists social_profile_ids bigint[] not null default '{}';

And i noticed this returning what i pointed originally with node-pg. This is reproduced even locally with pg 10.3.

Cool, thanks! Honestly two examples would be fine. I'd really just like to find another discussion to try to pin down how people end up with non-1-indexed results like this in "normal" pg use.

If everyone silently throws away indexes because their predecessors did, the production databases/queries that have been running for years are never fixed.

pg has other parsing limitations because of JS, e.g. bigint exceeds Number.MIN_SAFE_INTEGER so we return as strings. That's consistent across the type whereas this case is even less specific.

This lib does throw errors for unbalanced arrays but that's not something we ever expect for any valid query.

process.emitWarning is a softer option than throwing but I'm not sure how I feel about printing to stderr from a lib like this.

Yes i understand implications of usage int8 in JS. This is just what i have.

I did small research what happening in other drivers. Also this prefix it is dimensions information about slice, i am not sure why in my case PG consider whole column to be slice. This dimensions contains information what items selected from array.

psycorg2

They are just striping this https://github.com/psycopg/psycopg2/blob/6becf0ef550e8eb0c241c3835b1466eef37b1784/psycopg/typecast_array.c#L28-L48

and

https://github.com/psycopg/psycopg2/blob/6becf0ef550e8eb0c241c3835b1466eef37b1784/psycopg/typecast_array.c#L257-L258

They introduced this information in psycopg/psycopg2@e72f3db

ruby-pg

They also just skip it, but also check for validity

https://github.com/ged/ruby-pg/blob/b920bb33eaa4797cdfdca8885dd56dbd04d6b8ea/ext/pg_text_decoder.c#L406-L448

This is discussion about why they introduced this change https://bitbucket.org/ged/ruby-pg/issues/272

As i understand PG provides this information for e.g preallocating array and it is safe to skip. Because indexes could be negative. I do not think it is usefull for remapping this to any sort of indexes in array.

Cool, appreciate the research. Noticed some similar concerns (https://bitbucket.org/ged/ruby-pg/issues/272#comment-43874548) but I couldn't reproduce that comment about leading nils.

I'll go ahead and work a bit more on this soon, hopefully this week. I want to consider whether it makes sense to handle invalid inputs that we don't expect from pg here. Or more tests. But I'll definitely merge what you have here.

Thank you, appreciate it

Any updates?

Thanks for the reminder, let me merge this now

@bendrucker i think in future we can attach dimension information to array returned, as a separate property, if somebody will ask about it. Currently it does not looks like it is useful.

Yeah that sounds reasonable, probably can set it as a non-enumerable property or something without messing with the underlying data. I'll wait until someone asks for it since this whole area of functionality seems to just represent technical debt among lots of projects 🙂