rs_int_example failing

[f1ren]

Describe the bug
rs_int_example.py failing on An error occurred during each attempt of this action.

To Reproduce
Steps to reproduce the behavior:

1. ./run_docker.sh
2. python -m omnigibson.examples.action_primitives.rs_int_example

Expected behavior
Program ends after printing Finished executing place.

Screenshots

                  Plan has 2 steps
                  Executing arm movement plan step 1/2
                  Executing arm movement plan step 2/2
Traceback (most recent call last):
  File "/micromamba/envs/omnigibson/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/micromamba/envs/omnigibson/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/omnigibson/omnigibson/examples/action_primitives/rs_int_example.py", line 70, in <module>
    main()
  File "/root/omnigibson/omnigibson/examples/action_primitives/rs_int_example.py", line 60, in main
    execute_controller(controller.apply_ref(StarterSemanticActionPrimitiveSet.GRASP, apple), env)
  File "/root/omnigibson/omnigibson/examples/action_primitives/rs_int_example.py", line 18, in execute_controller
    for action in ctrl_gen:
  File "/root/omnigibson/omnigibson/action_primitives/starter_semantic_action_primitives.py", line 545, in apply_ref
    raise ActionPrimitiveErrorGroup(errors)
omnigibson.action_primitives.action_primitive_set_base.ActionPrimitiveErrorGroup: An error occurred during each attempt of this action.

Attempt 0: EXECUTION_ERROR: Could not navigate to the target position. Additional info: {'target pose': (tensor([ 0.0310, -0.6776,  0.0100]), tensor([ 0.0000,  0.0000, -0.8991,  0.4377]))}

Attempt 1: POST_CONDITION_ERROR: Grasp completed, but no object detected in hand after executing grasp. Additional info: {'target object': 'apple'}

Attempt 2: POST_CONDITION_ERROR: Grasp completed, but no object detected in hand after executing grasp. Additional info: {'target object': 'apple'}

Desktop (please complete the following information):

• OS: Ubuntu 24.04.1 LTS
• Isaac Sim Version (see docker)
• OmniGibson Version 1.1.1 (docker)

Additional context
The behavior is non-deterministic and different attempts are printed depending on state at failure.

[hang-yin]

Hi @f1ren , thanks for reaching out!

Since the action primitives use sampling-based motion planning, it is expected that the robot behavior can be non-deterministic. This primitives feature is also somewhat experimental and needs a lot more tuning. We are actively working on making this more robust and experimenting with alternative motion planners. But for now, you can try the following to make the primitives more fault-tolerant:

In omnigibson/examples/action_primitives/rs_int_example.py, you can look for controller.apply_ref(StarterSemanticActionPrimitiveSet.GRASP, apple) and controller.apply_ref(StarterSemanticActionPrimitiveSet.PLACE_ON_TOP, cabinet). This apply_ref function has an attempts argument, which defaults to 3. You can try to make this higher either 1) directly in the example you are running or 2) make the default higher in omnigibson/action_primitives/stater_semantic_action_primitives.py::StarterSemanticActionPrimitives::apply_ref.

We do have some really exciting progress, stay tuned for our next few releases!

[f1ren]

Thanks @hang-yin , setting attempts to 10 indeed seems to improve the results :)

Do you have a quick tip for resetting the planner state? It fails if I reset the env.
I'm trying to run it in a for loop, but the planner fails after a reset. I guess it holds the state before the reset.

[hang-yin]

Hi @f1ren , that's a good point! We don't currently have a reset method implemented (@cgokmen can confirm), but one hacky, and potentially slow way of doing this is you can manually create a new StarterSemanticActionPrimitives object after each environment reset. Improving the usability and robustness of these primitives is on our recent roadmap though, will probably have a better iteration of it in the upcoming months.

[f1ren]

Thanks for the tip @hang-yin , however, re-instantiation of StarterSemanticActionPrimitives
after env.reset() yields the same results.
Do you happen to know if there's any state that is stored somewhere else? Perhaps in the OMPL side?

Thanks again!

[hang-yin]

@cgokmen have you seen this type of behavior before?

[loubron23]

Hello author @hang-yin, I am currently using action primitives to try some simulation experiments, but I found that none of the three example codes' tasks can be completed. Sometimes the robot cannot grab objects because the arm joints cannot be planned, and sometimes it cannot reach the specified position. I executed it almost 50 times but only succeeded once. Is this normal?
Another question is that once it fails within 3 attempts, the entire program will be terminated with the following error. Is there a way to not terminate the program so that I can continue to control the robot to correct the error?

Traceback (most recent call last):
  File "/omnigibson-src/omnigibson/examples/action_primitives/test.py", line 87, in <module>
    main()
  File "/omnigibson-src/omnigibson/examples/action_primitives/test.py", line 82, in main
    execute_controller(controller.apply_ref(StarterSemanticActionPrimitiveSet.PLACE_ON_TOP, table), env)
  File "/omnigibson-src/omnigibson/examples/action_primitives/test.py", line 18, in execute_controller
    for action in ctrl_gen:
  File "/omnigibson-src/omnigibson/action_primitives/starter_semantic_action_primitives.py", line 545, in apply_ref
    raise ActionPrimitiveErrorGroup(errors)
omnigibson.action_primitives.action_primitive_set_base.ActionPrimitiveErrorGroup: An error occurred during each attempt of this action.

Attempt 0: EXECUTION_ERROR: Failed to place object at the desired place (probably dropped). The object was still released, so you need to grasp it again to continue. Additional info: {'dropped object': 'apple', 'target object': 'table'}

Attempt 1: PRE_CONDITION_ERROR: You need to be grasping an object first to place it somewhere.. Additional info: None

Attempt 2: PRE_CONDITION_ERROR: You need to be grasping an object first to place it somewhere.. Additional info: None
2024-12-17 07:31:39 [87,168ms] [Warning] [carb] [Plugin: omni.sensors.tiled.plugin] Module /isaac-sim/extscache/omni.sensors.tiled-0.0.4+106.0.0.lx64.r/bin/libomni.sensors.tiled.plugin.so remained loaded after unload request
2024-12-17 07:31:39 [87,171ms] [Warning] [carb] [Plugin: omni.spectree.delegate.plugin] Module /isaac-sim/kit/exts/omni.usd_resolver/bin/libomni.spectree.delegate.plugin.so remained loaded after unload request
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,173ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-12-17 07:31:39 [87,174ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Core' for removal
2024-12-17 07:31:39 [87,287ms] [Warning] [carb] Client omni.syntheticdata.plugin Failed to acquire interface [omni::graph::core::INode v4.10] while unloading all plugins
Fatal Python error: Segmentation fault

Current thread 0x00007f67ffac9740 (most recent call first):
  <no Python frame>

Extension modules: yaml._yaml, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, PIL._imaging, scipy._lib._ccallback_c, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._cythonized_array_utils, scipy.linalg._flinalg, scipy.linalg._solve_toeplitz, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_lapack, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.special._ellip_harm_2, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize.__nnls, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, psutil._psutil_linux, psutil._psutil_posix, lxml._elementpath, lxml.etree, _cffi_backend, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._statlib, scipy.stats._mvn, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._rcont.rcont, matplotlib._c_internal_utils, matplotlib._path, kiwisolver._cext, matplotlib._image, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, omni.mdl.pymdlsdk._pymdlsdk, PIL._imagingft, osqp._osqp, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, ompl.util._util, ompl.base._base, ompl.geometric._geometric (total: 178)
Segmentation fault (core dumped)