diff --git a/TASK_LIST.md b/TASK_LIST.md new file mode 100644 index 000000000..b40405baf --- /dev/null +++ b/TASK_LIST.md @@ -0,0 +1,420 @@ +# Task List + +TODO + +## Interactive Tasks + +TODO DOWNLOAD + +### Introduced in Evaluation 3 + +#### Retrieval - Containers + +TODO + +#### Retrieval - Obstacles + +TODO + +#### Retrieval - Occluders + +TODO + +### Introduced in Evaluation 4 + +#### Object Permanence (Interactive) + +TODO + +### Introduced in Evaluation 5 + +#### Agent Identification + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/d9f7e054-8d46-467a-8d16-dcf126af8a81 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/0b27f91e-9d74-482c-8e2f-236930c28bb0 + +Details: + +- You start on a tall platform bisecting the entire room. There will be an agent on one side of the room and a non-agent (a random object, like a sofa) on the other side of the room. As with our other interactive tasks, your goal is to “find and pickup the soccer ball”, but the soccer ball will not be visible. You will have to jump off the platform onto the side of the room containing the agent and request that the agent produce the soccer ball for you. +- The core goals of this task are learning: + - How to identify an agent based on surface features (like faces) and self-propulsion (like breathing and walking). + - To request help from an agent if the goal object (soccer ball) is not otherwise visible. +- In Eval 5, there will only be one agent in a scene, and the agent will always be able to produce the soccer ball upon request. However, these assumptions might change in future evals. +- Starting in release 0.5.3, there will be an InteractWithAgent action that facilitates the request for the agent to produce the soccer ball: + - The agent will rotate across multiple action steps to face you, reach behind its back to retrieve the soccer ball, and offer the ball to you. + - You can use the PickupObject action to pick up the ball from the agent’s hand. + - You will have to use the Pass action multiple times to wait for the agent to finish performing its animation/movement before you can pick up the ball. + - In future evals, the agent may instead refer you to the ball’s location via an animation like pointing. +- Starting in release 0.5.4, each agent can be configured to have a walking path. +- You will be able to walk across the full length of the platform before you jump off from it in order to verify that the soccer ball is not hiding behind any objects in the scene. + +#### Moving Target Prediction + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/22b4c78f-7eef-44a4-8e75-82306f12dc09 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/8b13cb6d-7fdd-48d9-94d4-9ba95ae6b073 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/aef99402-8029-476c-b431-6d931701fbe4 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/dd5b75e8-6612-4fb8-91e4-92a6e0217f43 + +Details: + +- The core goal of this task is to understand the speed and trajectory of the soccer ball in order to intercept it. At the start of the scene, the ball will immediately be rolled out from the “thrower” on one of the side walls (grey tube) at a speed that should give your AI enough time to think and retrieve it before it rolls into a pool of lava (and out of reach) positioned across the other side of the room. As in the real world, the ball is affected by physical properties like drag and friction. The ball will remain in contact with the floor during its movement (no bouncing). +- At the start of the scene, before the soccer ball is rolled out from the thrower, your AI will be forced to rotate in a complete circle (using 36 sequential RotateRight actions), so it may see the initial landscape of the room. +- Like other interactive scenes, you use the PickupObject action to retrieve the ball. +- You will start on a platform so you can get a good view of the entire room, but can walk off the platform at any time as normal. +- See “Navigation: Lava” for more information on adjusting our lava settings for training. + +#### Navigation - Holes + +Details: + +- If you move too close to a hole, you’ll fall in, and won’t be able to get out. +- Your AI will begin on a platform so it has a good view of the entire room before moving. + +#### Navigation - Lava + +Details: + +- Stepping too close to the lava will force you to immediately end the scene (you will only be allowed to use the “EndScene” action if you try to call the step function, or you can call end_scene yourself). This will be what happens during the evaluation. You can override the default setting using the steps_allowed_in_lava config property: https://nextcenturycorporation.github.io/MCS/install.html#steps-allowed-in-lava +- Stepping too close to the lava will give you an insurmountably large reward penalty. You will always receive this penalty, even if you must immediately end the scene due to the steps_allowed_in_lava setting (see above). You can override the default setting using the lava_penalty config property: https://nextcenturycorporation.github.io/MCS/install.html#lava-penalty +- Stepping too close to the lava will adjust the haptic_feedback and steps_in_lava properties of the StepMetadata output returned by that action step: haptic_feedback will be {"on_lava": true} and steps_in_lava will increase by one. https://nextcenturycorporation.github.io/MCS/api.html#machine_common_sense.StepMetadata +- Your AI will begin on a platform so it has a good view of the entire room before moving. + +#### Navigation - Ramps + +||| +|---|---| +![ramps_eval_5_ex_1](https://github.com/NextCenturyCorporation/MCS/assets/10994382/b2c87fde-ca3f-48cc-8ed4-597ceacc3d39) | ![ramps_eval_5_ex_3](https://github.com/NextCenturyCorporation/MCS/assets/10994382/33b9373a-77fd-4e9e-8050-88a9dd3a7a1c) + +Details: + +- Ramps will always be triangles/wedges. They can have angles up to 45 degrees. They will always be the same color/texture as their corresponding platform. +- Platforms can have either one or two levels. Each platform level will always have a different color/texture. Platforms will have “lips” (or edges) that surround their perimeter to make it impossible to move off the platform without using one of its corresponding ramps. + +#### Solidity + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/392d5b5e-977f-4c75-a169-7a50f5752cf7 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/308d2d09-e4af-472b-93f4-9f626692bdde + +Details: + +- You start on a tall platform bisecting the entire room. Notably, there is an extension of the platform coming out of either its left or right side, which is only accessible by walking on top of the platform. Additionally, the far end of the platform has "lips" (or edges) that surround the platform and make it impossible to move off the platform to the ground on that end. +- Each solidity scene contains 3 phases. You will be frozen (only able to perform Pass actions) at the start (about 60 to 100 action steps), but later you will be able to act freely. + - First, you will see a placer (pink cylinder) holding a soccer ball slowly descend from the ceiling and come to a stop either above the platform or in mid-air. Note that the placer hasn’t released its hold on the ball yet. + - Second, you will see a large occluder descend from the ceiling between you and the soccer ball blocking most (but not all) of your view of the far end of the room. This occluder will have three doors: one on ground level to the left of the platform; one on ground level to the right of the platform; and one in the middle at platform level. Once this “door occluder” has finished descending, it will no longer move. (This door occluder moves and works the same as in the Interactive Support Relations task.) + - Third, you will see the placer turn blue, indicating that it has released the soccer ball, and ascend back into the ceiling. However, due to the occluder, you can’t see the final position of the ball. You will then be unfrozen and must pickup the soccer ball to succeed in the trial. You will need to decide whether the ball has fallen to the ground on the left side (only accessible via the left door), the right side (only accessible via the right door), or the platform extension (only accessible via the middle door). +- A door can be opened using the OpenObject action. However, in these scenes, once you open one door, you will not be able to open another door. + +#### Spatial Elimination + +||| +|---|---| +![spatial_elimination_eval_5_ex_1](https://github.com/NextCenturyCorporation/MCS/assets/10994382/9152594e-ca9f-43eb-a499-7b107d7fe406) |![spatial_elimination_eval_5_ex_2](https://github.com/NextCenturyCorporation/MCS/assets/10994382/63df3f27-98df-46c0-888a-fc0887ae0708) +![spatial_elimination_eval_5_ex_3](https://github.com/NextCenturyCorporation/MCS/assets/10994382/83fa96da-fc7a-41c1-aa6a-42a4376443d8) |![spatial_elimination_eval_5_ex_4](https://github.com/NextCenturyCorporation/MCS/assets/10994382/d3897fdc-3fdf-408a-97ee-a0f6784f23eb) + +Details: + +- Like the Interactive Object Permanence tasks from Eval 4, these scenes contain a tall platform bisecting the entire room. You’re forced to move off the platform to either side, but, once you’ve made a choice, you can’t access the other side. + +#### Support Relations (Interactive Gravity Support) + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/436f1c93-8043-4599-adbc-e50dac0194ad + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/40caccb2-39d2-45ed-af55-8ed98657059b + +Details: + +- You start on a tall platform bisecting the entire room. Notably, the far end of the platform (beyond the door occluder) is raised to make it impossible to move forward on the platform past the middle door. +- Each support relations scene contains 3 phases. You will be frozen (only able to perform Pass actions) at the start (about 60 to 100 action steps), but later you will be able to act freely. + - First, you will see two placers (pink cylinders) holding a container slowly descend from the ceiling and come to a stop above the platform, so that the width of the container is either fully above the platform or partially above the platform. The container will always be open-topped (i.e. you don’t have to “Open” it), either symmetric or asymmetric (from your view point), and holding the soccer ball (clearly visible). Note that the placers haven’t released their hold on the container yet. + - Second, you will see a large occluder descend from the ceiling between you and the soccer ball blocking most (but not all) of your view of the far end of the room. This occluder will have three doors: one on ground level to the left of the platform; one on ground level to the right of the platform; and one in the middle at platform level. Once this “door occluder” has finished descending, it will no longer move. (This door occluder moves and works the same as in the Interactive Solidity task.) + - Third, you will see the placers turn blue, indicating that they have released the container, and ascend back into the ceiling. However, due to the occluder, you can’t see the final position of the container or the ball. You will then be unfrozen and must pickup the soccer ball to succeed in the trial. You will need to decide whether the container and the ball were properly supported by the platform (only accessible via the middle door) or were not supported by the platform and have therefore fallen to the ground on either the left side (only accessible via the left door) or the right side (only accessible via the right door). +- A door can be opened using the OpenObject action. However, in these scenes, once you open one door, you will not be able to open another door. + +#### Tools - Symmetric + +||| +|---|---| +![tool_use_eval_5_ex_1](https://github.com/NextCenturyCorporation/MCS/assets/10994382/e38e275b-99d3-4ce4-8851-28b6b1558d16) | ![tool_use_eval_5_ex_3](https://github.com/NextCenturyCorporation/MCS/assets/10994382/ebd8f106-e1d8-4db9-bb80-d3c79d386aca) + +Details: + +- Tools have their own object types that are all identical except in their length and width. They look like large grey rectangular prisms on wheels with a short cylinder sticking up from each corner. They have a unique grey hexagonal metallic texture. They are all low friction and light weight, to make them easy to push by force. https://nextcenturycorporation.github.io/MCS/schema.html#tool-objects +- The core goal of this task is for you to learn how to maneuver the tool object so it will collide with the soccer ball and cause it to roll from the lava into an accessible area of the room. To achieve this goal, we’ve prepared five actions for you to use with the tool object. All are acceptable to use during the evaluation. + - PushObject and PullObject, which apply a force on the center of a specific object, either directly away from you (push) or directly toward you (pull), based on your current facing. A large applied force may persist across multiple action steps. Some force will be transferred onto lighter objects (like the soccer ball) that are contacted by the pushed/pulled object. Note that smaller and lighter objects will realistically move more than larger or heavier objects under equal amounts of force. + - MoveObject, which moves a specific object forward, backward, left, or right by exactly 0.1 meter. This action does not apply any physical force. Lighter objects (like the soccer ball) that are contacted by the moved object will be moved out of the way in the appropriate direction. + - TorqueObject, which applies a rotational force on the center of a specific object, either clockwise or counter-clockwise. A large applied force may persist across multiple action steps. Note that smaller and lighter objects will realistically rotate more than larger or heavier objects under equal amounts of force. + - RotateObject, which rotates a specific object clockwise or counter-clockwise by exactly 5 degrees. This action does not apply any physical force. + - See the full action documentation here: https://nextcenturycorporation.github.io/MCS/api.html#machine_common_sense.Action +- See the section on the new lava task for more information about using lava. +- Please note: while only Pushing tools (or Moving them forward) will be necessary in Eval 5, future evals might require Pulling tools (or Moving them backward). + +### Introduced in Evaluation 6 + +#### Arithmetic and Number Comparison + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/57b6acc4-1cde-4e82-b56f-2aa23b8bced2 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/de4b01ea-5f48-46ac-83df-edd77776bcc0 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/87d6fe2a-97c6-4308-9340-70258245ae85 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/e2f4028f-7f2a-4004-9e22-967bc12c012f + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/fbe1d7a8-dfff-4001-97e0-4d7f8c40b372 + +Details: + +- You start on a platform bisecting the room (this is a forced choice task). One or more objects (soccer balls) are positioned on each side of the room. Like our other forced choice tasks, once you “hop off” the platform onto one side of the room, you cannot reach the other side. You must determine which side of the room contains more potential “target” objects, and pick up all of the objects on that side of the room. +- In Number Comparison tasks, you can move immediately. Nothing will change in the scene while it is running. +- In Arithmetic tasks, objects will either be added or subtracted from one side of the room. Immediately after the scene is loaded, one or more placers descend from the ceiling. The placers either hold new objects (soccer balls) and drop them on the floor, or pick up existing objects and raise them to the ceiling, making them unavailable. This may mean that the side of the room which had the most potential “target” objects when the scene was first loaded now no longer has the most. + - In Evaluation 6, scenes will not have both addition and subtraction – just one mathematical operation. + - In Evaluation 6, objects will only be added/subtracted on one side of the room. + - Sometimes all placers will remain empty, to represent addition/subtraction by zero. +- Some Arithmetic scenes will contain occluders, which are lowered into the room on both sides after the scene is loaded. You will be able to see the initial number of objects on each side, and you will be able to see how many objects are added to, or subtracted from, the scene, but you will not be able to see the final number of objects at rest before you are forced to “hop off” the platform (choose a side of the room). +- In both tasks, each side of the room may have between zero and five potential “target” objects. + +Notes: + +- These scenes use the new “multi retrieval” interactive goal category, to distinguish them from scenes with the existing “retrieval” category, which have only a single target. For more information on “multi retrieval” interactive goals, please see our API doc here: https://nextcenturycorporation.github.io/MCS/api.html#machine_common_sense.GoalCategory.MULTI_RETRIEVAL +- You will not receive a reward until all of the targets are picked up. +- In Arithmetic scenes, you will be restricted to only using Pass actions until the placers (and occluders) are finished moving. + +#### Imitation + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/1536702f-b2b6-4323-9da6-fc7af4bff486 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/cde91660-af4d-4289-91e5-689c38a59290 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/7a297753-d3c8-499a-8982-d3f2237ba146 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/526e2211-be7d-40cc-b62c-23ae09bd2bbe + +Details: + +- You start in a room looking at three containers lined up in a row. The containers are identical in size and shape (chest_1) but have different colors. After the scene begins, an agent approaches either one or two specific container(s) and opens them in a specific order, triggering a placer holding the target object (soccer ball) to descend from the ceiling, release the target object onto the floor, and ascend back into the ceiling. You are then kidnapped, during which the room is reset: the placer picks up the target object and raises it back up toward the ceiling, making the target object inaccessible; the containers are closed; and the agent moves out-of-the-way. You must remember the pattern the agent used to make the target object accessible, open the correct containers in the correct order, and pick up the target object when it is within your reach. +- The room always has three containers. Sometimes you need to open two containers; other times you only need to open one container. +- Sometimes you will be moved to a new location when you are kidnapped; other times you will remain in the same location. +- Sometimes the containers will be moved to a new location when you are kidnapped; sometimes they will be rotated; other times they will remain in the same location. If the containers are moved or rotated, they will always remain adjacent to each other, and in the same order relative to each other. + +Notes: + +- Imitation scenes use the new “imitation” interactive goal category to differentiate them from normal “retrieval” goals. For more information on “imitation” goals, please see our API doc here: https://nextcenturycorporation.github.io/MCS/api.html#machine_common_sense.GoalCategory.IMITATION +- You will be restricted to only using Pass actions until being kidnapped. While “frozen”, you will be able to see all of the containers, the agent, and the target object, once it has been released by the placer. +- The containers always start lined up on either the left side of your view, or the right side. The agent will approach the containers from the opposite side. +- Kidnapping and teleporting are performed using the EndHabituation action. Like other uses of this action, you will receive a blank frame during the kidnapping step. Unlike how this action worked in Eval 4, you will not receive any information (position/rotation) about the teleport destination (this information is now tracked internally by the MCS environment). +- Containers are opened using the OpenObject action, like in other scenes with containers. +- If you open an incorrect container, or you open the correct containers in an incorrect order, you will immediately be forced to end the scene without continuing further, thus failing the trial. + +#### Occluded Trajectory and Collisions (Interactive) + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/898d48b4-0c26-4a4c-9cf2-4de22e765059 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/4b2a5a02-24b3-49ef-a700-43abc37d56de + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/8b8bf840-52d0-4f13-a490-5f2c5b7f374f + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/97f421d2-eb72-4e1a-8437-b68d2e551d1e + +Details: + +- Interactive Trajectory – You start on a platform, and a strip of lava is bisecting the room (this is a forced choice task). A throwing device (or "thrower") is positioned across the room from you (within your initial view) holding the target object (soccer ball). Immediately after the scene begins, the thrower "launches" the target object so it starts to roll across the room. An occluding wall with two doors (or "two-door-occluder") descends from the ceiling while the target object is rolling, obstructing your view of the target object's complete trajectory and final position. You must determine whether the target object will end on either the left side or the right side of the room/lava, using your knowledge of its trajectory; then you must jump off from the platform on the correct side, walk up to the corresponding door, open it, find the target object, and pick it up. +- Interactive Collision – You start on a platform, and a strip of lava is bisecting the room (this is a forced choice task). A throwing device (or "thrower") is positioned across the room from you (within your initial view) holding a green ball, which is NOT the target object. The target object (soccer ball) is positioned elsewhere in the room (within your initial view), but not in the lava. Immediately after the scene begins, the thrower "launches" the green ball so it starts to roll across the room. The green ball either collides directly with the target object, hitting it "head-on", or completely misses the target object. An occluding wall with two doors (or "two-door-occluder") descends from the ceiling either immediately before or immediately after the collision, obstructing your view of the target's complete trajectory and final position. You must determine whether the target object will end on either the left side or the right side of the room/lava, using your knowledge of the green ball's trajectory and the possibility of a collision; then you must jump off from the platform on the correct side, walk up to the corresponding door, open it, find the target object, and pick it up. +- Sometimes the thrower will be oriented straight ahead, and the target object will remain on the same side of the room as it started; sometimes the thrower will be angled, and the target object will roll over the lava to the other side of the room. +- Sometimes the two-door-occluder will begin to descend immediately after the scene begins; sometimes it will wait a few steps, letting you see more of the target object's trajectory (or, in the Interactive Collisions task, the green ball's trajectory). +- Rolling over the lava does not affect the target object's movement any differently than rolling over the floor. Its movement is otherwise executed using our real-world physics simulation engine and is therefore affected by consistent physical properties like drag and friction. +- You can safely assume that the target object will never be launched with such a force or angle to cause it to "rebound" off a wall and cross over the lava to the other side of the room. +- If you step into the lava, you will be forced to immediately end the scene, thus failing it. Therefore, once you "jump off" the starting platform onto one side of the room, you will not have any way to cross over to the other side of the room. +- In the Interactive Collisions task, the green ball in the thrower will always appear the same. Nothing else in the room will be green. + +Notes: + +- You will be restricted to only using Pass actions until the door-occluder has finished descending. For more information on action restrictions, see the action_list property in the StepMetadata class: https://nextcenturycorporation.github.io/MCS/api.html#machine_common_sense.StepMetadata +- Stepping too close to the lava will force you to immediately end the scene (you will only be allowed to use the “EndScene” action if you try to call the step function, or you can call end_scene yourself). This will be what happens during the evaluation. You can override the default setting using the steps_allowed_in_lava config property: https://nextcenturycorporation.github.io/MCS/install.html#steps-allowed-in-lava +- Stepping too close to the lava will give you an insurmountably large reward penalty. You will always receive this penalty, even if you must immediately end the scene due to the steps_allowed_in_lava setting (see above). You can override the default setting using the lava_penalty config property: https://nextcenturycorporation.github.io/MCS/install.html#lava-penalty +- Stepping too close to the lava will adjust the haptic_feedback and steps_in_lava properties of the StepMetadata output returned by that action step: haptic_feedback will be {"on_lava": true} and steps_in_lava will increase by one. https://nextcenturycorporation.github.io/MCS/api.html#machine_common_sense.StepMetadata + +#### Set Rotation + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/b44fcc48-b7de-433a-b520-bf36371206ac + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/c293ea74-1f77-4985-8f23-8d73ec88e00d + +Details: + +- You start in a room looking at a turntable (cog). Between one and five identical open containers are positioned on top of the turntable so they can be seen clearly. After the scene begins, a placer holding the target object (soccer ball) descends from the ceiling and releases the target object into one of the containers. Then placers holding lids descend from the ceiling and release the lids, attaching one to the top of each container. Finally, the turntable rotates either 90, 180, 270, or 360 degrees, either clockwise or counter-clockwise, moving the containers along with it. You must determine which container holds the target object, walk up to that container, open it, and pick up the target object. +- When you open a container, the other containers will automatically become locked (this is a forced choice task). So if the first container you open is not the correct container, you will never be able to retrieve the target object, and will therefore fail the trial. +- In some scenes, instead of the turntable rotating, you will be forced to move around the turntable, to either the left or the right, so you end facing the turntable and containers from a different angle. Please see the Engineering Details below for more information. +- The turntable will only ever rotate the one time at the beginning. In scenes with forced movement, the turntable will never rotate. + +Notes: + +- You will be restricted to only using Pass actions until the turntable finishes rotating (or, in some scenes, until you are forced to move). While “frozen”, you will be able to clearly see all of the containers, the turntable, and the target object as it is placed into a container. +- In some scenes, forced movement will be performed by returning the action you must invoke in your next controller.step call (like "MoveLeft" or "MoveRight") via the StepMetadata.action_list property, in the same way you are forced to use only “Pass” actions during other specific steps (see the previous bullet). +- In Eval 6, the containers and lids in these scenes are always the same size, shape, and color. + - These containers can be opened using the OpenObject action, just like other containers. The lid will always open on the opposite side from where you are standing. + - These containers will be a consistent shape (cuboid) and color (green) for all Eval 6 scenes, which we hope will simplify the training of your systems. New shapes or colors may be introduced in Eval 7. + - Scenes with these containers will restrict you to use only Pass actions before the lid is placed onto the container, which happens at the beginning of the trial, before the “exploration” phase. + - Your teams are able to use the ILE Scene Generator to make scenes with these containers (either with the lid on them from the start of the scene, or with the lid placed on them by a placer during the course of the scene) – please see our ILE API documentation for details. +- Information about the turntables (giant grey rotating cogs): + - Turntables in evaluation scenes will rotate at a consistent speed of 5 degrees per action step, either clockwise or counter-clockwise. + - Objects and agents on top of a turntable (including the AI) will rotate along with the turntable. + - Turntables will be a consistent height, shape (cog), and color (grey) for all Eval 6 scenes, though their radii may change across different scenes. + - Scenes with turntables will restrict you to use only Pass actions while the turntable is rotating, which happens at the beginning of the trial, before the “exploration” phase. + +#### Shell Game + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/e0127554-36a6-4526-892c-bcc1341c2b7e + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/6a5ed2f0-b8b2-410f-a744-38cdab9b9a27 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/98730a4a-a201-4cdf-8162-218188c26ec2 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/107232ca-f28a-4cc0-85ff-22d189b61daa + +Details: + +- You start in a room looking at two or three identical open containers positioned in a line near each other. After the scene begins, a placer holding the target object (soccer ball) descends from the ceiling and releases the target object into one of the containers. Then placers holding lids descend from the ceiling and release the lids, attaching one to the top of each container. Finally, a placer descends from the ceiling, attaches itself to the container holding the target object, drags the container across the floor to a new location, and releases the container. This dragging motion always moves the container toward you, then sideways, then away from you, which sometimes occludes your view of the other container(s). You must determine which container holds the target object, walk up to that container, open it, and pick up the target object. +- When you open a container, the other containers will automatically become locked (this is a forced choice task). So if the first container you open is not the correct container, you will never be able to retrieve the target object, and will therefore fail the trial. +- In some scenes, the container will be dragged before the target object is placed into it and the containers' lids are attached. +- In some scenes, another placer will descend from the ceiling, attach itself to a second container (one that is not holding the target object), drag it across the floor to a new location, and release it. This means that two of the containers will have moved since the beginning of the scene. + +Notes: + +- You will be restricted to only using Pass actions until the lids are attached to their respective containers. While “frozen”, you will be able to clearly see all of the containers and the target object as it is placed into a container. +- In Eval 6, the containers and lids in these scenes are always the same size, shape, and color. + - These containers can be opened using the OpenObject action, just like other containers. The lid will always open on the opposite side from where you are standing. + - These containers will be a consistent shape (cuboid) and color (green) for all Eval 6 scenes, which we hope will simplify the training of your systems. New shapes or colors may be introduced in Eval 7. + - Scenes with these containers will restrict you to use only Pass actions before the lid is placed onto the container, which happens at the beginning of the trial, before the “exploration” phase. + - Your teams are able to use the ILE Scene Generator to make scenes with these containers (either with the lid on them from the start of the scene, or with the lid placed on them by a placer during the course of the scene) – please see our ILE API documentation for details. + +#### Spatial Reference + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/862044cd-7c33-4174-8677-f3aaacf691a6 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/31d8d9a4-7f22-4a21-bc0a-b5788ad88f2f + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/6400a0a9-e26e-4c64-9878-5f26394082f6 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/db675a03-0de2-4167-a9d7-a119d1e5a43a + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/e6d066e5-94ed-4287-bdfb-9ca796ded65d + +Details: + +- You start on a platform bisecting the room (this is a forced choice task). Each side of the room contains an identical closed container. One side of the room contains an agent who, after the scene begins, turns, walks toward, looks at, and points at the container secretly holding the target object (soccer ball). The agent keeps pointing at the container indefinitely. The other side of the room contains an unknown, faceless, blob-shaped entity (known as a “blob”) with a nose-like protrusion. The blob is positioned on a turntable (cog), and the turntable rotates so the blob’s “nose” is “pointing” at one of the containers, which may or may not be the same container at which the agent is pointing. Like our other forced choice tasks, once you “hop off” the platform onto one side of the room, you cannot reach the other side. You must determine which entity is the real agent, use the agent’s referential information (the direction of its walk, gaze, and point) to correctly identify the container holding the target object, walk up to that container, open it, and pick up the target object. +- The agent’s walk, gaze, and point will always be oriented in the same direction. This redundancy was introduced to make it easier for you to interpret the agent’s referential information. +- The container holding the target object will sometimes be on the side of the room with the agent, and other times be on the side of the room across from the agent. +- In some scenes, the agent will move and point before the turntable below the blob begins to rotate; in other scenes, the turntable will rotate fully before the agent begins to move. +- Some scenes will not have a blob – just an agent. +- For consistency, there is always a second turntable positioned underneath the agent, but it never rotates. Agents move on their own accord, but blobs cannot move by themselves. (This is another indication of agency.) + +Notes: + +- You will be restricted to only using Pass actions until the agent and turntable have stopped moving/rotating. While “frozen”, you will be able to clearly see the two containers, the agent, the blob, and the two turntables. +- Since the blobs are unknown entities by design, training scenes containing blobs will not be available. We have provided an example YAML configuration file for the ILE Scene Generator which uses everyday objects (like toy cars or wooden animals) in place of blobs. + +#### Spatial Reorientation + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/97fb77f4-9f35-41ee-9a7b-2efb649b8362 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/cf3e2b27-a4e0-4e1d-a165-425d9f1fc84b + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/27cfab0a-2b83-4fb5-b9af-665c1f05ed89 + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/a49184c7-211d-4c8d-870e-b0d286f7a4db + +Details: + +- You start on a platform bisecting the room (this is a forced choice task). Two identical open-topped containers are positioned in mirrored locations on each side of the room (the left and right). A placer holding the target (soccer ball) descends from the ceiling over one of the containers and drops it into the container. You are then “kidnapped” and teleported to either the front or the back of the room, but still standing on the platform. Like our other forced choice tasks, once you “hop off” the platform onto one side of the room, you cannot reach the other side. You must determine which side of the room contains the target, move over to it and pick it up. +- Most scenes provide cues so you can determine whether you have been teleported to the opposite side of the room: + - Sometimes either the left or right wall is a significantly different color than the other walls. + - Sometimes the room is shaped like a trapezoid: the left and right walls are angled inward, and either the front or back wall is shorter. + - Sometimes there is a large non-structural object, like a piece of furniture, on one side of the room; TA2 calls this an “unstable landmark.” In scenes containing both a stable landmark (differently colored room walls, or trapezoidal shaped rooms) and an unstable landmark (large objects like furniture that could theoretically be moved by an adult), your system should trust the location of the stable landmark more than the unstable landmark. + - It should be impossible to see inside the containers from your position on top of the platform due to the position of the containers and a barrier on the platform that stops you from moving too far forward. + +Notes: + +- Immediately after the scene is loaded, but before the placers start moving, you will be forced to rotate in a full circle (using the RotateRight action 36 times in a row), so you can see the complete layout of the room before you are kidnapped. Then, you will be restricted to only using Pass actions until the placers are finished moving, and after you have been kidnapped. +- Kidnapping and teleporting are performed using the EndHabituation action. Like other uses of this action, you will receive a blank frame during the kidnapping step. Unlike how this action worked in Eval 4, you will not receive any information (position/rotation) about the teleport destination (this information is now tracked internally by the MCS environment). +- Whenever you are kidnapped, you we be teleported a little “off-center”, regardless of whether you remain on the same side of the room (near the “back” wall) or the opposite side (near the “front” wall). For example, if your original position is (X=0, Z=-7.5), your new position may be (X=0.1, Z=-7.6) + +#### Tools - Asymmetric and Tool Choice + +||| +|---|---| +![tool_choice_ex_1](https://github.com/NextCenturyCorporation/MCS/assets/10994382/20668ccc-bffd-4b3d-9fdc-f775766687b3) | ![tool_hooked_ex_1](https://github.com/NextCenturyCorporation/MCS/assets/10994382/01587ca1-3e64-4a9b-8a03-5415fd26fb5b) + +Details: + +- Asymmetric Tool + - The Asymmetric Tool task is a new variant of the Tool Use task from Eval 5. Rather than pushing a symmetric tool (a.k.a. “rectangular-shaped tool”), you must pull an asymmetric tool (a.k.a. “hooked tool” or “L-shaped tool”) to extract the target object (soccer ball) from the middle of a pool of lava (by pulling the tool so it comes into contact with the soccer ball and causes the ball to roll out from the lava to a safe, reachable area). + - The pool of lava in Asymmetric Tool scenes always extends out to three of the exterior walls of the room, so you will not be able to extract the target object by only pushing the tool – you must also learn how to pull the tool, and how to identify situations in which doing so is necessary. +- Tool Choice + - The Tool Choice task is similar to the Tool Use task from Eval 5, but in a forced choice paradigm. You start on a platform bisecting the room. Each side of the room contains a pool of lava surrounding a soccer ball. One side of the room has a tool that can be used to extract the soccer ball from the pool of lava in order to pick it up (by pushing the tool so it comes into contact with the soccer ball and causes the ball to roll out from the lava to a safe, reachable area). The soccer ball on the other side of the room is inaccessible. Like our other forced choice tasks, once you “hop off” the platform onto one side of the room, you cannot reach the other side. You must determine which side of the room contains the “useful tool”, correctly use the tool to extract the soccer ball from the pool of lava, and pick up the soccer ball. + - The other side of the room may contain: no tool; a tool that is too small and thus cannot be used to successfully extract the soccer ball from the pool of lava; a tool that is broken into multiple pieces which are too small; or a tool that is completely separated from the soccer ball by an obstruction and thus cannot reach the soccer ball. + +Notes: +- Asymmetric Tool + - To pull an object, you may use either the PullObject action, which applies a force to the object using the environment’s physics engine, or the MoveObject action, which moves the object by an exact amount (0.1 meter) in a specified direction, bypassing the physics engine completely. Both actions will be valid during the evaluation. + - Like the original Tool Use task, the asymmetric tool may need to be rotated (using the RotateObject and/or TorqueObject actions) before it can be used effectively. +- Tool Choice + - For scoring purposes, only the soccer ball on the side of the room containing the “useful tool” is considered the “target object”. + - In Eval 6, the “useful tool” in Tool Choice scenes will always be a symmetric (rectangular-shaped) tool (not an asymmetric tool). + +### Introduced in Evaluation 7 + +#### Hidden Set Rotation + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/0b22f8c2-6774-4cc0-a4e6-9fc0b0b3201b + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/1868299b-f26a-4692-8cfd-63dafe25557a + +Details: + +- This task is setup exactly like the Eval 6 Set Rotation task, but a giant “tube occluder” descends from the ceiling to hide the turntable’s rotation. Your goal is still to find and pick up the target object. +- For scenes in which the turntable would rotate, the tube occluder goes down just before the turntable begins to rotate (after the lids are placed onto the containers). The tube occluder goes up again after enough time for the turntable to rotate 360 degrees (even if the turntable rotates less than that). +- For scenes in which you are forced to move around the turntable, the tube occluder goes down just before the forced movement (after the lids are placed onto the containers). The tube occluder goes up again after enough time for your agent to circumnavigate the turntable a full 360 degrees (even if your agent moves less than that). +- You can assume the only change hidden by the tube occluder is the turntable rotating between 0 and 360 degrees. +- When you open a container, the other containers will automatically become locked. So if the first container you open is not the correct container, you will never be able to retrieve the target object, and will therefore fail the trial. + +#### Knowledgeable Agents + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/6fe0e811-b53b-4171-95dc-c0496440379b + +https://github.com/NextCenturyCorporation/MCS/assets/10994382/0afc8420-f068-4767-904e-b3afc87621d9 + +Details: + +- You begin positioned in the center of the back of the room, looking forward. A platform bisects the room (this is a forced choice task), and the target object (soccer ball) starts visible on top of the platform. An open container is positioned on each side of the platform, as well as a simulation-controlled agent. Each agent is facing either toward you (and can therefore see both containers) or away from you (and cannot see either container). +- Immediately after the scene begins, the two simulation-controlled agents rotate 180 degrees, reversing their orientations, and remain in these orientations until they turn to point (after the occluder rises up to the ceiling – see below). This rotation should help you identify which way the agent is facing (i.e. the front of their head versus the back). +- After this rotation, an occluder descends from the ceiling to hide your view of the containers and the target object. The simulation-controlled agents are on the same side of the occluder as the containers and target object. Importantly, the occluder is short enough so you can always see the faces of the two agents, and therefore you always know whether each agent can see the containers (because it's facing toward you) or cannot see the containers (because it’s facing away from you). +- Then, two “placers” (blue and pink poles) descend from the ceiling, one after another, and appear to pick up the target object, carry it across to one of the containers, and release the target object into that container. One of these placers is a decoy, but you cannot see which placer actually picked up the target object, or which container it was dropped into, due to the occluder. However, any simulation-controlled agents facing toward you can see the placers, containers, and target object. +- Next, two more placers descend from the ceiling and attach lids to the containers, and the occluder rises up to the ceiling, so you can see the two closed containers. +- Finally, each simulation-controlled agent turns and points to the container they believe holds the target object. Your goal, like in all interactive tasks, is to pick up the target object. To achieve your goal, you must determine which agent is knowledgeable (based on your understanding of whether that agent could see where the target object was placed), determine which container the knowledgeable agent is pointing at, move to that container, open it, and pick up the target object. Like our other forced choice tasks, once you “hop off” the platform onto one side of the room, you cannot reach the other side. +- Sometimes the knowledgeable agent will be on the platform furthest away from you, and sometimes the knowledgeable agent will be in the pit and therefore closer to you. + +Notes: + +- You will be “frozen” (restricted to only using Pass actions) and thus unable to move until after the simulation-controlled agents point. +- Using the InteractWithAgent action on an agent while it's pointing will not provide you with any additional information. + +#### Tools - Secondary Tool Use + +TODO + +Details: + +- You begin positioned randomly in a room containing two sections of lava. The first section of lava surrounds the target object (soccer ball). The second section of lava surrounds a hooked (L-shaped) tool. There is also a rectangular tool somewhere in the room. Neither the target object nor the hooked tool are within your reach due to the lava. Your goal, like in all interactive tasks, is to pick up the target object. To achieve your goal, you must know how to use the rectangular tool to push the hooked tool out from the lava, then use the hooked tool to pull the target object out from the lava. +- The position of the target object in the room, and the amount of lava surrounding it, will make it impossible for you to use the rectangular tool alone to push the target object out from the lava (and bypass using the hooked tool). +- The rectangular tool may start in proper alignment with the hooked tool, or may require you to move and/or rotate it into alignment. +- The hooked (L-shaped) tool may have two sides with equal lengths, or two sides with different lengths. + +Notes: + +- Like in previous Evals, you can use the PushObject, PullObject, and MoveObject actions to move tools, and the TorqueObject and RotateObject actions to rotate tools. Pushing, Pulling, and Moving one tool into a second tool will cause that second tool to move along with it as you would expect (please note that some bugs for this were fixed in MCS release version 0.7.0). +- Tools may have novel colors/textures applied to them (including, but not limited to, the colors/textures used in Eval 6), but the general shape of the tools will remain the same. diff --git a/docs/source/dev.rst b/docs/source/dev.rst index ccafd733f..fa72d1d32 100644 --- a/docs/source/dev.rst +++ b/docs/source/dev.rst @@ -81,6 +81,12 @@ Sphinx Documentation - Sphinx: https://www.sphinx-doc.org/en/master/ - Sphinx's own Tutorial: https://www.sphinx-doc.org/en/master/usage/quickstart.html +To generate the Sphinx documentation locally, from the docs folder: + +.. code-block:: console + + (mcs) $ make html + Python Style Guide ------------------ @@ -151,14 +157,6 @@ team Team name identifier to prefix to scene history and video files (default: ''). -timeout -^^^^^^^ - -(int, optional) - -If the amount of time between steps exceeds the amount specified (represented in seconds), throw and error and attempt to end scene (default: 3600 (or 1 hour)). - - Handling Pull Requests From Contributors ---------------------------------------- diff --git a/docs/source/index.rst b/docs/source/index.rst index 61924f69e..c43386716 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -12,7 +12,6 @@ Machine Common Sense Documentation api scenes schema - training models containerize remotely diff --git a/docs/source/install.rst b/docs/source/install.rst index dfcb51331..bc07c4951 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -121,33 +121,33 @@ To use a specific configuration, you can either pass in a file path or dictionar Config File Properties ********************** -only_return_goal_object -^^^^^^^^^^^^^^^ +controller_timeout +^^^^^^^^^^^^^^^^^^ -(boolean, optional) +(int, optional) -If `true`, only returns the goal object from the object list output in the metadata. Metadata Tier will override `only_return_goal_object`. Default: false +Sets the time (in seconds) to allow on controller initialization before timing out. Default 600 -disable_position +disable_depth_maps ^^^^^^^^^^^^^^^ (boolean, optional) -If `true`, does not generate position information output in the metadata. Metadata Tier will override `disable_position`. Default: false +If `false`, will generate depth maps. Metadata Tier will override `disable_depth_maps`. Will only generate depth maps for Metadata tier [level1, level2, oracle]. Default: false -disable_depth_maps +disable_object_masks ^^^^^^^^^^^^^^^ (boolean, optional) -If `false`, will generate depth maps. Metadata Tier will override `disable_depth_maps`. Will only generate depth maps for Metadata tier [level1, level2, oracle]. Default: false +If `false`, will generate object masks. Metadata Tier will override `disable_depth_maps`. Will only generate object masks for Metadata tier [level2, oracle]. Default: false -disable_object_masks +disable_position ^^^^^^^^^^^^^^^ (boolean, optional) -If `false`, will generate object masks. Metadata Tier will override `disable_depth_maps`. Will only generate object masks for Metadata tier [level2, oracle]. Default: false +If `true`, does not generate position information output in the metadata. Metadata Tier will override `disable_position`. Default: false goal_reward ^^^^^^^^^^^^^^^ @@ -184,8 +184,16 @@ The `metadata` property describes what metadata will be returned by the MCS Pyth - `none`: Only returns the images (but no depth maps or object masks), camera info, and properties corresponding to the player themself (like head tilt). No information about specific objects will be included. If no metadata level is set: + - `default`: Fallback if no metadata level is specified. Only meant for use during development (evaluations will never be run this way). Includes metadata for visible and held objects in the scene, as well as camera info and properties corresponding to the player. Does not include depth maps or object masks. +only_return_goal_object +^^^^^^^^^^^^^^^ + +(boolean, optional) + +If `true`, only returns the goal object from the object list output in the metadata. Metadata Tier will override `only_return_goal_object`. Default: false + steps_allowed_in_lava ^^^^^^^^^^^^^^^ @@ -228,20 +236,6 @@ step_penalty Changes the negative penalty recieved for every step. Default: 0.001 -timeout -^^^^^^^ - -(int, optional) - -Sets the time (in seconds) to allow on a single step before timing out. Default 3600 - -controller_timeout -^^^^^^^^^^^^^^^^^^ - -(int, optional) - -Sets the time (in seconds) to allow on controller initialization before timing out. Default 600 - terminal_output ^^^^^^^^^^^^^^^ @@ -257,6 +251,13 @@ The terminal output mode controls what output metadata is logged to the terminal - `scene`: Prints metadata for the scene's room, including all structural objects. - `false` or `none`: Prints no terminal output, besides initialization, warning, and error messages. +timeout +^^^^^^^ + +(int, optional) + +Sets the time (in seconds) to allow on a single step before timing out. Default 3600 + top_down_camera ^^^^^^^^^^^^^^^ diff --git a/docs/source/remotely.rst b/docs/source/remotely.rst index 4c48cddf6..15069b6fa 100644 --- a/docs/source/remotely.rst +++ b/docs/source/remotely.rst @@ -2,8 +2,9 @@ Running Remotely ================ Requirements: -* Make sure your system has Python 3.7 or greater. -* Make sure MCS has been installed per the install instructions + +- Make sure your system has Python 3.7 or greater. +- Make sure MCS has been installed per the install instructions To run MCS on a remote GPU server, use the following steps to launch an X11 server. @@ -23,7 +24,8 @@ To run MCS on a remote GPU server, use the following steps to launch an X11 serv PCI BusID : PCI:0:30:0 # Create the Xserver configuration using the BusID - $ sudo nvidia-xconfig --use-display-device=None --virtual=600x400 --output-xconfig=/etc/X11/xorg.conf --busid=PCI:0:30:0 + # Your virtual display size should be much larger than the MCS Unity window size (default 600x400) + $ sudo nvidia-xconfig --use-display-device=None --virtual=1200x800 --output-xconfig=/etc/X11/xorg.conf --busid=PCI:0:30:0 # Launch Xserver $ sudo /usr/bin/Xorg :0 & @@ -53,7 +55,7 @@ The following code was run on an AWS p2-xlarge with the Ubuntu Deep Learning AMI .. code-block:: bash # Follow the instructions listed above to launch Xserver - $ sudo nvidia-xconfig --use-display-device=None --virtual=600x400 --output-xconfig=/etc/X11/xorg.conf --busid=PCI:0:30:0 + $ sudo nvidia-xconfig --use-display-device=None --virtual=1200x800 --output-xconfig=/etc/X11/xorg.conf --busid=PCI:0:30:0 $ sudo /usr/bin/Xorg :0 & # Download the MCS repository diff --git a/docs/source/scenes.rst b/docs/source/scenes.rst index 384360005..798e0c15c 100644 --- a/docs/source/scenes.rst +++ b/docs/source/scenes.rst @@ -1,51 +1,10 @@ Example Scenes ============== -:doc:`Documentation ` - Interactive Scenes ------------------ -Eval 5 Example Interactive Scenes -********************************* - -Relevant for the Summer 2022 evaluation (Eval 5): - -:download:`holes_eval_5_ex_1.json <./scenes/holes_eval_5_ex_1.json>` - -:download:`holes_eval_5_ex_2.json <./scenes/holes_eval_5_ex_2.json>` - -:download:`lava_eval_5_ex_1.json <./scenes/lava_eval_5_ex_1.json>` - -:download:`lava_eval_5_ex_2.json <./scenes/lava_eval_5_ex_2.json>` - -:download:`ramps_eval_5_ex_1.json <./scenes/ramps_eval_5_ex_1.json>` - -:download:`ramps_eval_5_ex_2.json <./scenes/ramps_eval_5_ex_2.json>` - -:download:`ramps_eval_5_ex_3.json <./scenes/ramps_eval_5_ex_3.json>` - -:download:`ramps_eval_5_ex_4.json <./scenes/ramps_eval_5_ex_4.json>` - -:download:`solidity_eval_5_ex_1.json <./scenes/solidity_eval_5_ex_1.json>` - -:download:`solidity_eval_5_ex_2.json <./scenes/solidity_eval_5_ex_2.json>` - -:download:`spatial_elimination_eval_5_ex_1.json <./scenes/spatial_elimination_eval_5_ex_1.json>` - -:download:`spatial_elimination_eval_5_ex_2.json <./scenes/spatial_elimination_eval_5_ex_2.json>` - -:download:`spatial_elimination_eval_5_ex_3.json <./scenes/spatial_elimination_eval_5_ex_3.json>` - -:download:`spatial_elimination_eval_5_ex_4.json <./scenes/spatial_elimination_eval_5_ex_4.json>` - -:download:`tool_use_eval_5_ex_1.json <./scenes/tool_use_eval_5_ex_1.json>` - -:download:`tool_use_eval_5_ex_2.json <./scenes/tool_use_eval_5_ex_2.json>` - -:download:`tool_use_eval_5_ex_3.json <./scenes/tool_use_eval_5_ex_3.json>` - -:download:`tool_use_eval_5_ex_4.json <./scenes/tool_use_eval_5_ex_4.json>` +For more interactive scenes, please see TODO Playroom ******** @@ -61,18 +20,14 @@ An open room containing over 40 objects for undirected exploration. Retrieval Goal ************** -Soccer ball retrieval goal for the Fall 2021 evaluation (Eval 4): +Soccer ball retrieval goal: :download:`retrieval_goal_example_with_soccer_ball.json <./scenes/retrieval_goal_example_with_soccer_ball.json>` -Silver trophy retrieval goal for the Winter 2020 evaluation (Eval 3): - -:download:`retrieval_goal_example_with_trophy.json <./scenes/retrieval_goal_example_with_trophy.json>` - Hinged Containers ***************** -Soccer ball inside hinged containers for the Fall 2021 evaluation (Eval 4): +Soccer ball hidden inside hinged containers: .. list-table:: @@ -80,273 +35,15 @@ Soccer ball inside hinged containers for the Fall 2021 evaluation (Eval 4): :download:`hinged_container_example_with_soccer_ball.json <./scenes/hinged_container_example_with_soccer_ball.json>` -Silver trophy inside hinged containers for the Winter 2020 evaluation (Eval 3): - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/hinged_container_example_with_trophy.gif - - :download:`hinged_container_example_with_trophy.json <./scenes/hinged_container_example_with_trophy.json>` - -Interactive Object Permanence and Reorientation Tasks -***************************************************** - -The room has different dimensions/bounds, and isn't necessarily square. Previously, the room's dimensions were always [-5, 5] on both the X and the Z axes. - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_12_by_8.gif - - :download:`template_12_by_8.json <./scenes/template_12_by_8.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_2_by_4.gif - - :download:`template_2_by_4.json <./scenes/template_2_by_4.json>` - -The room's outer walls are individually, distinctly colored. Previously, all of the room's outer walls were always the same color. - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_individually_colored_walls.gif - - :download:`template_individually_colored_walls.json <./scenes/template_individually_colored_walls.json>` - -The performer agent is positioned on top of a flat, raised platform. Moving off the edge of the platform will cause the performer agent to automatically, instantaneously fall down to the floor, and the performer agent will not be able to move back on top of the platform. - -.. list-table:: +Passive Agent Scenes +-------------------- - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_platform_independent.gif - - :download:`template_platform_independent.json <./scenes/template_platform_independent.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_platform_bisecting.gif - - :download:`template_platform_bisecting.json <./scenes/template_platform_bisecting.json>` - -The performer agent is temporarily "frozen" (can only use the Pass action) at the start and/or in the middle of an interactive scene. This is done by the same method that is used for the passive/VoE scenes (see the StepMetadata.action_list property). - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_frozen_first_10_steps.gif - - :download:`template_frozen_first_10_steps.json <./scenes/template_frozen_first_10_steps.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_frozen_after_10_steps.gif - - :download:`template_frozen_after_10_steps.json <./scenes/template_frozen_after_10_steps.json>` - -A cylindrical mechanism attached to a wall or the ceiling throws (for interactive object permanence) or drops (for reorientation) the target object (i.e. soccer ball) into the scene. - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_mechanism_dropping_soccer_ball_v2.gif - - :download:`template_mechanism_dropping_soccer_ball_v2.json <./scenes/template_mechanism_dropping_soccer_ball_v2.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_mechanism_throwing_soccer_ball.gif - - :download:`template_mechanism_throwing_soccer_ball.json <./scenes/template_mechanism_throwing_soccer_ball.json>` - -The performer agent is "kidnapped" (can only call the EndHabituation action) and teleported to another position in the current room one or more times. Prior to being kidnapped, the performer agent is able to move around and explore its environment for a limited number of steps. On the kidnapped step, the returned images will be black. Prior to the final kidnapping, StepMetadata.habituation_trial will be an integer; after the final kidnapping, StepMetadata.habituation_trial will be "None" to denote the test trial. - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_kidnapping.gif - - :download:`template_kidnapping.json <./scenes/template_kidnapping.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_kidnapping_then_frozen.gif - - :download:`template_kidnapping_then_frozen.json <./scenes/template_kidnapping_then_frozen.json>` - -Combination of multiple elements. For example: a platform, being frozen, and a mechanism throwing the soccer ball into the scene. - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/template_interactive_object_permanence_scene.gif - - :download:`template_interactive_object_permanence_scene.json <./scenes/template_interactive_object_permanence_scene.json>` - -End-to-end scenes. VALIDATION ONLY. - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/eval_4_intobjperm_validation_01.gif - - :download:`eval_4_intobjperm_validation_01.json <./scenes/eval_4_intobjperm_validation_01.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/eval_4_reor_validation_01.gif - - :download:`eval_4_reor_validation_01.json <./scenes/eval_4_reor_validation_01.json>` - -Passive/Intuitive Physics Scenes --------------------------------- - -Gravity Support: Objects Falling Down -************************************* - -All of these examples are PLAUSIBLE - -.. list-table:: +For more passive agent scenes, please see TODO - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_01.gif - - :download:`gravity_support_ex_01.json <./scenes/gravity_support_ex_01.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_02.gif - - :download:`gravity_support_ex_02.json <./scenes/gravity_support_ex_02.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_03.gif - - :download:`gravity_support_ex_03.json <./scenes/gravity_support_ex_03.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_04.gif - - :download:`gravity_support_ex_04.json <./scenes/gravity_support_ex_04.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_05.gif - - :download:`gravity_support_ex_05.json <./scenes/gravity_support_ex_05.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_06.gif - - :download:`gravity_support_ex_06.json <./scenes/gravity_support_ex_06.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_07.gif - - :download:`gravity_support_ex_07.json <./scenes/gravity_support_ex_07.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_08.gif - - :download:`gravity_support_ex_08.json <./scenes/gravity_support_ex_08.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_09.gif - - :download:`gravity_support_ex_09.json <./scenes/gravity_support_ex_09.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_10.gif - - :download:`gravity_support_ex_10.json <./scenes/gravity_support_ex_10.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_11.gif - - :download:`gravity_support_ex_11.json <./scenes/gravity_support_ex_11.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/gravity_support_ex_12.gif - - :download:`gravity_support_ex_12.json <./scenes/gravity_support_ex_12.json>` - - -Object Permanence and Spatio-Temporal Continuity: Objects Moving on Multiple Axes Behind Occluders -************************************************************************************************** - -Relevant for the Fall 2021 evaluation (Eval 4). Objects may move on only the X axis (as in previous evaluations), on both the X and Z axes (see the "move deep" example scenes), and/or on both the X and Y axes (see the "move toss" example scenes). - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_deep_fast_01.gif - - :download:`move_deep_fast_01.json <./scenes/move_deep_fast_01.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_deep_fast_02.gif - - :download:`move_deep_fast_02.json <./scenes/move_deep_fast_02.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_deep_fast_03.gif - - :download:`move_deep_fast_03.json <./scenes/move_deep_fast_03.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_deep_fast_04.gif - - :download:`move_deep_fast_04.json <./scenes/move_deep_fast_04.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_toss_fast_01.gif - - :download:`move_toss_fast_01.json <./scenes/move_toss_fast_01.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_toss_fast_02.gif - - :download:`move_toss_fast_02.json <./scenes/move_toss_fast_02.json>` - -Object Permanence: Objects Moving on Multiple Axes and Stopping Behind Occluders -******************************************************************************** - -Relevant for the Fall 2021 evaluation (Eval 4). Objects may move across the entire screen and exit on the other side (as in previous evaluations), or come to a natural stop behind the occluder. Objects may move on only the X axis (as in previous evaluations), on both the X and Z axes (see the "move deep" example scenes), and/or on both the X and Y axes (see the "move toss" example scenes). - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_slow_01.gif - - :download:`move_slow_01.json <./scenes/move_slow_01.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_slow_02.gif - - :download:`move_deep_slow_01.json <./scenes/move_deep_slow_01.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_deep_slow_01.gif - - :download:`move_toss_slow_01.json <./scenes/move_toss_slow_01.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_deep_slow_02.gif - - :download:`move_slow_02.json <./scenes/move_slow_02.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_toss_slow_01.gif - - :download:`move_deep_slow_02.json <./scenes/move_deep_slow_02.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/move_toss_slow_02.gif - - :download:`move_toss_slow_02.json <./scenes/move_toss_slow_02.json>` - - -Object Permanence: Objects Falling Down Behind Occluders -******************************************************** - -Relevant for the Winter 2020 evaluation (Eval 3). - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/object_permanence_plausible.gif - - :download:`object_permanence_plausible.json <./scenes/object_permanence_plausible.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/object_permanence_implausible.gif - - :download:`object_permanence_implausible.json <./scenes/object_permanence_implausible.json>` - -Spatio-Temporal Continuity: Objects Moving Across Behind Occluders -****************************************************************** - -Relevant for the Winter 2020 evaluation (Eval 3). - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/spatio_temporal_continuity_plausible.gif - - :download:`spatio_temporal_continuity_plausible.json <./scenes/spatio_temporal_continuity_plausible.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/spatio_temporal_continuity_implausible.gif - - :download:`spatio_temporal_continuity_implausible.json <./scenes/spatio_temporal_continuity_implausible.json>` - -Agents Scenes -------------- - -Agents Have Goals and Preferences -********************************* - -.. list-table:: - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/agents_preference_expected.gif - - :download:`agents_preference_expected.json <./scenes/agents_preference_expected.json>` - - * - .. figure:: https://mcs-documentation.s3.amazonaws.com/images/agents_preference_unexpected.gif - - :download:`agents_preference_unexpected.json <./scenes/agents_preference_unexpected.json>` +Passive Physics Scenes +---------------------- +For more passive physics scenes, please see TODO Simple Scenes ------------- @@ -362,7 +59,6 @@ With Objects :download:`block_close.json <./scenes/block_close.json>` - With Walls ********** diff --git a/docs/source/training.rst b/docs/source/training.rst deleted file mode 100644 index 53d28a61d..000000000 --- a/docs/source/training.rst +++ /dev/null @@ -1,78 +0,0 @@ -Training Datasets -================= - -Evaluation 4 - Fall 2021 --------------------------- - -Passive Agent -************* - -Subtasks: - -- Single object scenes (~10K), just like Eval 3 -- Object preference scenes (~10K), just like Eval 3 -- Multiple agents scenes (4K), new to Eval 4 -- Instrumental action scenes (4K), new to Eval 4 - -JSON scene configuration files: - -- https://eval-4-data.s3.amazonaws.com/eval_4_agent_training_dataset.zip - -Rendered videos: - -- https://nyu-datasets.s3.amazonaws.com/agent_instrumental_action_training_videos.zip -- https://nyu-datasets.s3.amazonaws.com/agent_multiple_agents_training_videos.zip -- https://nyu-datasets.s3.amazonaws.com/agent_object_preference_training_videos.zip -- https://nyu-datasets.s3.amazonaws.com/agent_single_object_training_videos.zip - -Passive Intuitive Physics -************************* - -Please generate your own training datasets using our Scene Generator software here: - -- https://github.com/NextCenturyCorporation/mcs-scene-generator - -Interactive -*********** - -For the container, obstacle, and occluder tasks, please generate your own training datasets using our Scene Generator software here: - -- https://github.com/NextCenturyCorporation/mcs-scene-generator - -For the new interactive object permanence and reorientation tasks, please generate your own training datasets using our example scene templates here: - -https://github.com/NextCenturyCorporation/MCS/tree/master/machine_common_sense/scenes#interactive-object-permanence-and-reorientation-tasks - -Evaluation 3 - Winter 2020 --------------------------- - -*Please use the most recent 0.3.X release version* - -Passive Agent (only expected scenes): - -- https://evaluation-training-scenes.s3.amazonaws.com/eval3/training-single-object.zip -- https://evaluation-training-scenes.s3.amazonaws.com/eval3/training-object-preference.zip - -Passive Intuitive Physics (only plausible scenes): - -- https://evaluation-training-scenes.s3.amazonaws.com/eval3/training-passive-physics.zip - -Example Scenes: -- https://github.com/NextCenturyCorporation/MCS/tree/master/machine_common_sense/scenes - -Evaluation 2 - Summer 2020 --------------------------- - -*Use a release version between 0.0.9 and 0.1.0* - -Interactive: - -https://evaluation2-training-scenes.s3.amazonaws.com/interaction-scenes.zip - -Passive IntPhys: - -https://evaluation2-training-scenes.s3.amazonaws.com/intphys-scenes.zip - -Passive IntPhys Validation Data: - -https://evaluation2-training-scenes.s3.amazonaws.com/validation-intphys-scenes.zip \ No newline at end of file