Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@carderne
Copy link
Contributor

@carderne carderne commented Nov 5, 2021

Without pyarrow installed, I got the following error:

2021-11-05 16:10:20.389 | INFO     | __main__:stac:33 - using satextractor.stac.gcp_region_to_item_collection stac creator.
Error executing job with overrides: []
Traceback (most recent call last):
  File "venv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 62, in _call_target
    return _target_(*args, **kwargs)
  File "venv/lib/python3.9/site-packages/satextractor/stac/stac.py", line 47, in gcp_region_to_item_collection
    df = get_sentinel_2_assets_df(client, region, start_date, end_date)
  File "venv/lib/python3.9/site-packages/satextractor/stac/stac.py", line 168, in get_sentinel_2_assets_df
    dfs.append(query_job.to_dataframe())
  File "venv/lib/python3.9/site-packages/google/cloud/bigquery/job/query.py", line 1644, in to_dataframe
    return query_result.to_dataframe(
  File "venv/lib/python3.9/site-packages/google/cloud/bigquery/table.py", line 1938, in to_dataframe
    record_batch = self.to_arrow(
  File "venv/lib/python3.9/site-packages/google/cloud/bigquery/table.py", line 1713, in to_arrow
    raise ValueError(_NO_PYARROW_ERROR)
ValueError: The pyarrow library is not installed, please install pyarrow to use the to_arrow() function.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sat-extractor/src/satextractor/cli.py", line 185, in main
    stac(cfg)
  File "sat-extractor/src/satextractor/cli.py", line 44, in stac
    item_collection = hydra.utils.call(
  File "venv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 180, in instantiate
    return instantiate_node(config, *args, recursive=_recursive_, convert=_convert_)
  File "venv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 249, in instantiate_node
    return _call_target(_target_, *args, **kwargs)
  File "venv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 64, in _call_target
    raise type(e)(
  File "venv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 62, in _call_target
    return _target_(*args, **kwargs)
  File "venv/lib/python3.9/site-packages/satextractor/stac/stac.py", line 47, in gcp_region_to_item_collection
    df = get_sentinel_2_assets_df(client, region, start_date, end_date)
  File "venv/lib/python3.9/site-packages/satextractor/stac/stac.py", line 168, in get_sentinel_2_assets_df
    dfs.append(query_job.to_dataframe())
  File "venv/lib/python3.9/site-packages/google/cloud/bigquery/job/query.py", line 1644, in to_dataframe
    return query_result.to_dataframe(
  File "venv/lib/python3.9/site-packages/google/cloud/bigquery/table.py", line 1938, in to_dataframe
    record_batch = self.to_arrow(
  File "venv/lib/python3.9/site-packages/google/cloud/bigquery/table.py", line 1713, in to_arrow
    raise ValueError(_NO_PYARROW_ERROR)
ValueError: Error instantiating 'satextractor.stac.stac.gcp_region_to_item_collection' : The pyarrow library is not installed, please install pyarrow to use the to_arrow() function.

@carderne
Copy link
Contributor Author

carderne commented Nov 5, 2021

DLQ fixed following #10

@carderne
Copy link
Contributor Author

carderne commented Nov 5, 2021

And added exponential backoff

@carderne carderne mentioned this pull request Nov 5, 2021
@carderne carderne changed the title Add PyArrow dependency DLQ, Pyarrow, backoff Nov 5, 2021
@carderne
Copy link
Contributor Author

carderne commented Nov 5, 2021

Feel free to pick and choose!

@carderne carderne mentioned this pull request Nov 5, 2021
@frandorr frandorr merged commit f5c937f into FrontierDevelopmentLab:main Nov 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants