Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Doc] Set action_spec instead of input_spec#1657

Merged
vmoens merged 8 commits intopytorch:mainfrom
FrankTianTT:main
Nov 3, 2023
Merged

[Doc] Set action_spec instead of input_spec#1657
vmoens merged 8 commits intopytorch:mainfrom
FrankTianTT:main

Conversation

@FrankTianTT
Copy link
Contributor

@FrankTianTT FrankTianTT commented Oct 26, 2023

Description

input_spec is protected and does not support with direct setting. Example code in CEM and MPPI documentation will cause error:

Traceback (most recent call last):
  File "/Users/frank/Projects/target-domain-fast-adaptation/tdfa/mpc/try.py", line 50, in <module>
    env = MyMBEnv(world_model)
  File "/Users/frank/opt/anaconda3/envs/tdfa/lib/python3.8/site-packages/torchrl/envs/common.py", line 137, in __call__
    instance: EnvBase = super().__call__(*args, **kwargs)
  File "/Users/frank/Projects/target-domain-fast-adaptation/tdfa/mpc/try.py", line 19, in __init__
    self.input_spec = CompositeSpec(
  File "/Users/frank/opt/anaconda3/envs/tdfa/lib/python3.8/site-packages/torchrl/envs/common.py", line 315, in __setattr__
    return super().__setattr__(key, value)
  File "/Users/frank/opt/anaconda3/envs/tdfa/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1754, in __setattr__
    super().__setattr__(name, value)
  File "/Users/frank/opt/anaconda3/envs/tdfa/lib/python3.8/site-packages/torchrl/envs/common.py", line 427, in input_spec
    raise RuntimeError("input_spec is protected.")
RuntimeError: input_spec is protected.

Let's set the action_spec and let the program decide the input_spec.

Motivation and Context

Why is this change required? What problem does it solve?
If it fixes an open issue, please link to the issue here.
You can use the syntax close #15213 if this solves the issue #15213

  • I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds core functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation (update in the documentation)
  • Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

  • I have read the CONTRIBUTION guide (required)
  • My change requires a change to the documentation.
  • I have updated the tests accordingly (required for a bug fix or a new feature).
  • I have updated the documentation accordingly.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 26, 2023
vmoens
vmoens reviewed Oct 27, 2023
View reviewed changes
Copy link
Collaborator

@vmoens vmoens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!
The remaining field in input spec should be placed in state_spec which can be edited.

self.state_spec = CompositeSpec(...)

@vmoens vmoens added the Documentation Improvements or additions to documentation label Oct 27, 2023
@FrankTianTT
Copy link
Contributor Author

FrankTianTT commented Oct 27, 2023

@vmoens I have corrected it, if we are on the same page:)

@FrankTianTT
Copy link
Contributor Author

FrankTianTT commented Oct 27, 2023

I have corrected it, if we are on the same page:)

@vmoens Sorry, I just checked and directly changing self.observation_spec to self.state_spec will cause new problems:

Traceback (most recent call last):
  File "/Users/frank/Projects/target-domain-fast-adaptation/tdfa/mpc/try2.py", line 54, in <module>
    planner = MPPIPlanner(
  File "/Users/frank/opt/anaconda3/envs/tdfa/lib/python3.8/site-packages/torchrl/modules/planners/mppi.py", line 136, in __init__
    super().__init__(env=env, action_key=action_key)
  File "/Users/frank/opt/anaconda3/envs/tdfa/lib/python3.8/site-packages/torchrl/modules/planners/common.py", line 37, in __init__
    in_keys = list(env.observation_spec.keys(True, True))
  File "/Users/frank/opt/anaconda3/envs/tdfa/lib/python3.8/site-packages/torchrl/envs/common.py", line 1159, in observation_spec
    observation_spec = self.output_spec["full_observation_spec"]
  File "/Users/frank/opt/anaconda3/envs/tdfa/lib/python3.8/site-packages/torchrl/data/tensor_specs.py", line 3271, in __getitem__
    return self._specs[idx_unravel]
KeyError: 'full_observation_spec'

This seems to be because MPCPlannerBase expects an observation_spec that can normalize the input to the algorithm. In the example code of ModelBasedEnvBase, MyMBEnv initializes both observation_spec and state_spec:

...         self.observation_spec = CompositeSpec(
...             hidden_observation=UnboundedContinuousTensorSpec((4,))
...         )
...         self.state_spec = CompositeSpec(
...             hidden_observation=UnboundedContinuousTensorSpec((4,)),
...         )

Actually this caused me more doubts, are the state_spec and observation_spec above the same thing? What I mean is that ModelBasedEnvBase is a stateless environment, and its step should accept $s_t, a_t$ and return $s_{t+1}, r_{t+1}$. At this time, whether state_spec corresponds to $s_t$ and observation_spec corresponds to $s_{t+1}$?

More specifically, if I want to use ModelBasedEnvBase to learn an environment (a stateful environment, such as gym or dmc), should its observation_spec and state_spec both be set to the observation_spec of the environment being learned, In order to support both model learning and MPC at the same time?

vmoens
vmoens reviewed Oct 30, 2023
View reviewed changes
@FrankTianTT FrankTianTT mentioned this pull request Nov 2, 2023
Open
3 tasks
vmoens
vmoens approved these changes Nov 3, 2023
View reviewed changes
Copy link
Collaborator

@vmoens vmoens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

@vmoens vmoens merged commit ce8a1c1 into pytorch:main Nov 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vmoens vmoens vmoens approved these changes

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@FrankTianTT @vmoens @facebook-github-bot