feat(spare/tableau): Ingest, flatten, and upload Spare resources to tableau #547

huangh · 2025-08-01T13:05:20Z

Prototype import spare to dev
Import all spare to dev/prod

Related Devops/Terraform:
https://github.com/mbta/terraform_modules/pull/287
https://github.com/mbta/terraform_modules/pull/289
https://github.com/mbta/terraform_modules/pull/295

https://github.com/mbta/devops/pull/3042

Why do we need changes?

Spare/Paratransit tech is reusing the existing Tableau architecture developed by LAMP to upload data to tableau for analysts. all spare data is in the /spare partition of the same dataplatform buckets, and require specific permissions to access (as implemented in the devops tickets above.

What changes does this PR propose?

The core work here is to generically convert and flatten the files into a tableau hyperapi compliant shape. A simple way to do this is to simply convert everything to strings, ~~but here we added a utility to flatten/explode/convert all fields into a fully flat structure.~~ Based on Comments, implemented just string flattening as a reasonable default. This is reusable and generic for other Tableau uploading tasks.

We also autogenerated the input/output and processing methods for each of these inputs, so adding a new source is simply adding it to a whilelist, and it will get processed and uploaded.

How were these changes validated?

Tested locally, tested main conversion method, tested in staging (since there is no environment to test in via dev...)

What questions should reviewers consider?

Limitations:

We removed the ability to do "custom" conversions for now as they were not necessary, but an improvement in the future may be to add includes/excludes, or as requested leave structs as just a "string" if it causes analysis issues by being too exploded/deep.

Another thing to watch is the runtime of this, as we've added ~46 new resources to upload, and this all runs in the Tableau cron job - which runs every hour.

github-actions · 2025-08-26T17:48:56Z

LCOV of commit `a95c7be` during Continuous Integration (Python) #1244

Summary coverage rate:
  lines......: 74.9% (2577 of 3440 lines)
  functions..: 32.9% (191 of 580 functions)
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ingestion/config_busloc_trip.py                                        |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/gtfs_rt_detail.py                                            |88.2%     17| 0.0%     8|    -      0
  src/lamp_py/ingestion/utils.py                                                     |47.0%    115|18.2%    22|    -      0
  src/lamp_py/runtime_utils/process_logger.py                                        | 100%     64|50.0%    12|    -      0

…rom s3, extracts the schema, and then renames fields that are invalid for tableau, and returns that schema

…_schema" that operates on schemas and not tables. this might not be used...or be possible. wip

…rform operations like flatten. this is generic, can do anything

…hat dont upload to tableau, but need to address

…tableau_schema method

…bility

…s nailed down

…estrain

…ment all the column types to handle

github-actions · 2025-08-26T18:26:39Z

LCOV of commit `4731eba` during Continuous Integration (Python) #1246

Summary coverage rate:
  lines......: 74.7% (2569 of 3440 lines)
  functions..: 32.9% (191 of 580 functions)
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ingestion/config_busloc_trip.py                                        |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/gtfs_rt_detail.py                                            |88.2%     17| 0.0%     8|    -      0
  src/lamp_py/ingestion/utils.py                                                     |47.0%    115|18.2%    22|    -      0
  src/lamp_py/runtime_utils/process_logger.py                                        | 100%     64|50.0%    12|    -      0

…ns - not used just yet.

github-actions · 2025-08-27T01:48:44Z

LCOV of commit `0b4c262` during Continuous Integration (Python) #1247

Summary coverage rate:
  lines......: 74.7% (2569 of 3440 lines)
  functions..: 32.9% (191 of 580 functions)
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ingestion/config_busloc_trip.py                                        |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/gtfs_rt_detail.py                                            |88.2%     17| 0.0%     8|    -      0
  src/lamp_py/ingestion/utils.py                                                     |47.0%    115|18.2%    22|    -      0
  src/lamp_py/runtime_utils/process_logger.py                                        | 100%     64|50.0%    12|    -      0

huangh · 2025-08-27T13:42:41Z

src/lamp_py/tableau/jobs/lamp_jobs.py

break out lamp jobs from spare jobs to stay organized

… logic

github-actions · 2025-08-27T13:57:20Z

LCOV of commit `2d4d31f` during Continuous Integration (Python) #1248

Summary coverage rate:
  lines......: 74.7% (2569 of 3440 lines)
  functions..: 32.9% (191 of 580 functions)
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ingestion/config_busloc_trip.py                                        |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/gtfs_rt_detail.py                                            |88.2%     17| 0.0%     8|    -      0
  src/lamp_py/ingestion/utils.py                                                     |47.0%    115|18.2%    22|    -      0
  src/lamp_py/runtime_utils/process_logger.py                                        | 100%     64|50.0%    12|    -      0

github-actions · 2025-08-27T15:57:37Z

LCOV of commit `47d290e` during Continuous Integration (Python) #1250

Summary coverage rate:
  lines......: 74.6% (2613 of 3504 lines)
  functions..: 32.6% (193 of 592 functions)
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ingestion/config_busloc_trip.py                                        |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/gtfs_rt_detail.py                                            |88.2%     17| 0.0%     8|    -      0
  src/lamp_py/ingestion/utils.py                                                     |47.0%    115|18.2%    22|    -      0
  src/lamp_py/runtime_utils/process_logger.py                                        | 100%     64|50.0%    12|    -      0
  src/lamp_py/tableau/__init__.py                                                    |35.0%     20| 0.0%     6|    -      0
  src/lamp_py/tableau/spare/default_converter.py                                     |65.9%     44|33.3%     6|    -      0

tests/tableau/test_convert_types.py

github-actions · 2025-08-27T19:40:36Z

LCOV of commit `8daaaee` during Continuous Integration (Python) #1251

Summary coverage rate:
  lines......: 74.4% (2615 of 3516 lines)
  functions..: 32.4% (192 of 592 functions)
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ingestion/config_busloc_trip.py                                        |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/gtfs_rt_detail.py                                            |88.2%     17| 0.0%     8|    -      0
  src/lamp_py/ingestion/utils.py                                                     |47.0%    115|18.2%    22|    -      0
  src/lamp_py/runtime_utils/process_logger.py                                        | 100%     64|50.0%    12|    -      0
  src/lamp_py/tableau/__init__.py                                                    |35.0%     20| 0.0%     6|    -      0
  src/lamp_py/tableau/spare/default_converter.py                                     |55.4%     56|16.7%     6|    -      0

skyqrose

I haven't reviewed the code in depth, but based on talking to you out of band and a spot check of the data in Tableau, 👍.

Thanks for taking care of this, I would not have been able to do it myself.

huangh force-pushed the hh-250729-feat-spare branch from 126f959 to 82e3697 Compare August 14, 2025 17:41

huangh added 28 commits August 26, 2025 14:15

allow None for num_days to mean all days in create_tableau_parquet

e7cdf30

add new get_default_tableau_schema_from_s3 method that grabs a file f…

66b5f30

…rom s3, extracts the schema, and then renames fields that are invalid for tableau, and returns that schema

rename flatten_schema to flatten_table_schema, and add a new "flatten…

b0d3237

…_schema" that operates on schemas and not tables. this might not be used...or be possible. wip

rename method fallout

8f4e016

refactor jobs into lamp jobs and spare jobs - add spare job for vehicles

ee40485

update runner

f02574d

add new remote_files for spare resources

5b93766

update filtered_hyper to allow preprocess method on a dataframe to pe…

4116e2c

…rform operations like flatten. this is generic, can do anything

spare conversion method to flatten. dropping some columns right now t…

d19a102

…hat dont upload to tableau, but need to address

update output_processed_schema for bus_recent to utilize new default_…

7fb00f7

…tableau_schema method

split up spare jobs parts to parquet and hyper again for future flexi…

d6f3a91

…bility

move flatten to spare jobs to consolidate before the file structure i…

9a7ca60

…s nailed down

black/cleanup

ad3bf96

add a combined parquet/hyper create method to call from the spare job

2c33fcc

rename combined job

cac278f

remove flatten_schema - unused method

d8448a7

add docstrings

ea7f690

remove comment

c42625d

blintype

808f92d

format exceptions so they show up in 1 record in splunk

5247002

no longer need to_frame with updated polars library

d2a898c

get uuid properly

14c60b5

only print out last 12 chars of uuid identifier to make the reduce ey…

0537661

…estrain

revert - print full uuid

461877b

ignore regenerate_bus_metric since its requiring new permissions in aws

49d2a2d

spare autogen to grab all schemas and size the tableau convert - docu…

5f66327

…ment all the column types to handle

to_frame() no longer needed with polars upgrade

96137ba

logger comment

88256db

huangh added 3 commits August 26, 2025 14:15

disable test_convert_types not ready

343754e

add more folders and lints to ignore

0a1ea83

black, pylint, mypy

25ef982

huangh force-pushed the hh-250729-feat-spare branch from a95c7be to 25ef982 Compare August 26, 2025 18:15

correct spare tableau project location

4731eba

huangh added 4 commits August 26, 2025 21:39

update autogen to not spit out the schema reading and printing portio…

46e32b2

…ns - not used just yet.

flesh out spare runner with options

0f5aeb0

handle u64 -> i64 because tableau does not support

e30612c

dont lint autogen

0b4c262

huangh commented Aug 27, 2025

View reviewed changes

src/lamp_py/tableau/jobs/lamp_jobs.py

Copy link

Collaborator Author

huangh Aug 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

break out lamp jobs from spare jobs to stay organized

huangh added 2 commits August 27, 2025 09:46

remove bus convert types - prototype code for the final autoconverter…

62f7a13

… logic

add note to add timezone logic to default_converter

2d4d31f

huangh added 2 commits August 27, 2025 11:22

add test for convert_to_tableau_flat_schema

5e0f01b

move pipeline getting into method to speed up module import

47d290e

huangh changed the title ~~Hh 250729 feat spare~~ feat(spare/tableau): Ingest, flatten, and upload Spare resources to tableau Aug 27, 2025

huangh marked this pull request as ready for review August 27, 2025 16:02

huangh requested review from ealexa05 and skyqrose August 27, 2025 16:03

skyqrose reviewed Aug 27, 2025

View reviewed changes

tests/tableau/test_convert_types.py Outdated Show resolved Hide resolved

huangh added 2 commits August 27, 2025 15:34

modify to convert lists and structs to strings instead

96f7698

blick

8daaaee

huangh requested a review from skyqrose August 27, 2025 19:40

skyqrose approved these changes Aug 28, 2025

View reviewed changes

huangh merged commit 06a8aed into main Aug 28, 2025
6 checks passed

huangh deleted the hh-250729-feat-spare branch August 28, 2025 15:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(spare/tableau): Ingest, flatten, and upload Spare resources to tableau #547

feat(spare/tableau): Ingest, flatten, and upload Spare resources to tableau #547

Uh oh!

huangh commented Aug 1, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 26, 2025

Uh oh!

github-actions bot commented Aug 26, 2025

Uh oh!

github-actions bot commented Aug 27, 2025

Uh oh!

huangh Aug 27, 2025

Uh oh!

github-actions bot commented Aug 27, 2025

Uh oh!

github-actions bot commented Aug 27, 2025

Uh oh!

Uh oh!

github-actions bot commented Aug 27, 2025

Uh oh!

skyqrose left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat(spare/tableau): Ingest, flatten, and upload Spare resources to tableau #547

feat(spare/tableau): Ingest, flatten, and upload Spare resources to tableau #547

Uh oh!

Conversation

huangh commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why do we need changes?

What changes does this PR propose?

How were these changes validated?

What questions should reviewers consider?

Uh oh!

github-actions bot commented Aug 26, 2025

LCOV of commit a95c7be during Continuous Integration (Python) #1244

Uh oh!

github-actions bot commented Aug 26, 2025

LCOV of commit 4731eba during Continuous Integration (Python) #1246

Uh oh!

github-actions bot commented Aug 27, 2025

LCOV of commit 0b4c262 during Continuous Integration (Python) #1247

Uh oh!

huangh Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 27, 2025

LCOV of commit 2d4d31f during Continuous Integration (Python) #1248

Uh oh!

github-actions bot commented Aug 27, 2025

LCOV of commit 47d290e during Continuous Integration (Python) #1250

Uh oh!

Uh oh!

github-actions bot commented Aug 27, 2025

LCOV of commit 8daaaee during Continuous Integration (Python) #1251

Uh oh!

skyqrose left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

huangh commented Aug 1, 2025 •

edited

Loading

LCOV of commit `a95c7be` during Continuous Integration (Python) #1244

LCOV of commit `4731eba` during Continuous Integration (Python) #1246

LCOV of commit `0b4c262` during Continuous Integration (Python) #1247

LCOV of commit `2d4d31f` during Continuous Integration (Python) #1248

LCOV of commit `47d290e` during Continuous Integration (Python) #1250

LCOV of commit `8daaaee` during Continuous Integration (Python) #1251