This pull request introduces 2 alerts when merging fd272ce into e8bf86e - view on LGTM.com

new alerts:

• 1 for Useless assignment to local variable
• 1 for Dereferenced variable may be null

Here is a notebook sharing results: https://colab.research.google.com/drive/1GSIF78B62hNskyr-SYtUMgMdsZ7tJpJQ?usp=sharing

I can think of use cases where you would want that old behavior as an option

Can you provide some? I can't think of any reason one would want to move the top-down camera and the agent at the same time, especially when it means the images from the top-down camera will no longer capture the entire scene.

Now ready to merge.
See https://colab.research.google.com/drive/1GSIF78B62hNskyr-SYtUMgMdsZ7tJpJQ?usp=sharing for all results.

Pushed a fix for FloorPlan309, where the camera was actually being placed below the scene. This was because the Ceiling in this room (and in none of the others) actually extends below the floor.

Hi @mattdeitke

Thanks for this new feature!

Since I was using ai2thor==3.1.0 (on Windows), how can I merge this new feature and make it work in ai2thor-3.1.0?

I tried to git clone ai2thor==3.1.0 and manually change the code files needed and pip install -e ., but that does not work and report error "ValueError: Invalid action: GetMapViewCameraProperties".

Thanks!

Just a follow-up, since I'm on Windows, does that mean I need to rebuild everything from ai2thor/unity folder in Unity to make this new feature work? The new feature is related to the change of the controller, so I guess the answer is yes?

Thanks!

Hey @xubo92,

Yes, if you rebuild locally from the latest commit on the main branch, it will all work, even with version 3.1.0 of the Python API.

Hi @mattdeitke

I tried this new feature and found out it only works for single agent. When I set agentCount more than 1, then the third_party_camera_frames is empty. Is this normal? Can it by any chance support multi-agent case?

Thanks!

And a follow-up question for this feature: how can I enable this top-down view all the time for real-time visualization while running controller for many steps?

I tried this code https://colab.research.google.com/drive/1GSIF78B62hNskyr-SYtUMgMdsZ7tJpJQ?usp=sharing#scrollTo=CvZ2PkYp95U2 , it works good when I use plt to show one image. But when I only run my controller and want to visualize in a top-down view constantly, it does not work and still using first-person view.

Thanks!

I tried this new feature and found out it only works for single agent. When I set agentCount more than 1, then the third_party_camera_frames is empty. Is this normal? Can it by any chance support multi-agent case?

Here is a demo of it working in the multi-agent case: https://colab.research.google.com/drive/1C_-9zqAqhSUcZvrxVlwlTMZQzWnEzY9k?usp=sharing.

One needs to also pass in agentId and then look at the event that corresponds to that id.

how can I enable this top-down view all the time for real-time visualization while running controller for many steps?

How are you visualizing the frames?

For the single agent case, check out plot_frames in ai2thor-colab. It doesn't yet work with the MultiAgentEvent, but I can probably extend it to if that'd be useful.

Hi @mattdeitke

Thanks a lot for these examples!

Probably I'm not saying clearly. Here is a code snippet I use

// initialize a controller
self.controller = Controller(agentMode=self.agent_types[0], agentCount=2, scene=self.scene, width=1600, height=900, local_executable_path=local_executable_path)

// setup top-down view
event = self.controller.step(action="GetMapViewCameraProperties")
event = self.controller.step(action="AddThirdPartyCamera", agentId=0, **event.metadata["actionReturn"])

// after setting top-down view, for each move, I expect a unity windows pop out showing what happened in the scene (multiple agents moving), in a top-down view
for i in range(100):
act = np.random.choice(ACTS)
self.controller.step(action=act, agentId=0)
self.controller.step(action=act, agentId=1)

Basically, I'm running a RL algorithm using ai2thor with multiple agents moving around. I expect to visualize the simulated scenes continuously, in a top-down view, so that I can have a sense of the policy training process by keeping an eye on the movement of agents. I expect the top-down view can give me a view of two (or) more agents in a room (not like the first-person view that I have to choose the agentId and get its local view.)

Not sure how to achieve this.

Thanks!

I remember when use the old command "ToggleMapView", it works like a switch, and one only needs to call it somewhere once (initialization) and after that the default view is switched to top-down view. I thought the new feature works the same way, especially when one has multiple agents so that top-down view can give a whole scene containing every agent.

Sorry I have a lot of questions here...

I thought the new feature works the same way

I see. You're looking at the Unity window instead of the image frames.

Basically, the old ToggleMapView took the agents camera, and just move it to the ceiling. But, this had many problems (described here: #599) and breaks nearly all interaction and a lot of the metadata.

Instead, what is now happening, is that we just add a new camera to the scene (a third party camera) and it is positioned such that it captures the scene from a top-down view. However, the agent's camera will still show up in the Unity window.

I don't think it's easy to change which camera is displayed in the Unity window.

What I would do is take videos of the top-down sequences, and save them locally. So, you're code may become:

# initialize a controller
self.controller = Controller(agentMode=self.agent_types[0], agentCount=2, scene=self.scene, width=1600, height=900, local_executable_path=local_executable_path)

# setup top-down view
event = self.controller.step(action="GetMapViewCameraProperties")
event = self.controller.step(action="AddThirdPartyCamera", agentId=0, **event.metadata["actionReturn"])

# after setting top-down view, for each move, I expect a unity windows pop out showing what happened in the scene (multiple agents moving), in a top-down view
map_frames = []
for i in range(100):
    act = np.random.choice(ACTS)
    e1 = self.controller.step(action=act, agentId=0)
    e2 = self.controller.step(action=act, agentId=1)
    map_frames.append(e1.events[0].third_party_camera_frames[0])
    map_frames.append(e2.events[0].third_party_camera_frames[0])

# save the video
from moviepy.editor import ImageSequenceClip
clip = ImageSequenceClip(frames, fps=25)
clip.write_videofile("trajectory.mp4")

Hi @mattdeitke

Thank you so much for clarifying the difference! Now I understand what is happening.

Saving frames and create a video sounds good! I'll try it.

Thanks!

Hi@mattdeitke
I wonder if it possible to achieve that when I control the robot walking in the room. And the same time,I have the second window to show the total scene of the room (for observe the motion of the robot)

Hi @yaofeng11,

There isn't currently a way to view the images from 2 separate cameras in separate Unity windows. But, what might be a workaround is you can do is save images or videos from the agent's perspective and from the top-down perspective, and then view them afterwards.

I just put this notebook demo together on that, which you may find useful:
https://colab.research.google.com/drive/1p6A_LDZA853-6boLdo4ha_1Bg6Cy3Jj9?usp=sharing

Let me know if that helps,
Matt :)