feature/union-data #71
Conversation
fivetran-jamie
left a comment
There was a problem hiding this comment.
Still testing but wanted to get this stuff back to you
| @@ -0,0 +1,76 @@ | |||
| {% macro github_union_connections(connection_dictionary, single_source_name, single_table_name, default_identifier) %} | |||
There was a problem hiding this comment.
Should we remove the default_identifier argument? It was necessary for Netsuite since the Netsuite1 vs Netsuite2 names are slightly different (which is why the table names != the source table names), but here the single_table_name and default_identifier are always the same
I don't think it's necessary for the other packages we plan to roll union_data out to as well (I think)
There was a problem hiding this comment.
From standup, we decided to use default_identifier=single_table_name so we can keep this and for another package to use if necessary. We'll store macros like this in a central area but just for reference and not as a package.
fivetran-jamie
left a comment
There was a problem hiding this comment.
Looking good but some requested changes!
Co-authored-by: Jamie Rodriguez <[email protected]>
| loader: Fivetran | ||
| config: | ||
| loaded_at_field: _fivetran_synced | ||
| enabled: "{{ var('github_sources', []) == [] }}" |
There was a problem hiding this comment.
We also need to include this in the table-level enabled configs that include variables, like on line 22 for issue_assignee
config:
loaded_at_field: _fivetran_synced
enabled: "{{ var('github__using_issue_assignee', true) and var('github_sources', []) == [] }}"Otherwise the enabled config gets overwritten and some github source tables are included in the DAG
Co-authored-by: Jamie Rodriguez <[email protected]>
…github into feature/union-data merge
PR Overview
Package version introduced in this PR:
This PR addresses the following Issue/Feature(s):
Summary of changes:
Submission Checklist
*.*_union_connectionsmacro call{{ *.apply_source_relation() }}source_relationto final select listenabled: "{{ var('github_sources', []) == [] }}"to src.ymlsource_relationcolumn to:<package_name>_sources: []to varssource_relationwas addedChangelog