Update morpheus dev container to support flag for building with debug python build + source files.

[drobison00]

resolves #21

Add the ability to build the morpheus dev container with a debug build of cpython + source libraries.

[drobison00]

@mdemoret-nv @pdmack Please take a look when you're able.

[mdemoret-nv]

I love this feature but I dont think it needs to be included in our development docker image like it currently is. This is hopefully going to be a very rarely used tool that makes more sense as a post development build stage to make it opt-in.

My comments can be summarized by:

1. Add a section to ci/conda/recipes/run_conda_build.sh to run the recipe similar to the other recipes in that script
2. Make a script for installing the built conda package to clean up the Dockerfile and allow the debug python package to be installed outside of a container (for example, somewhere in ci/conda/recipes/python-dbg should work)
3. Move all of the Dockerfile changes to a post-development build stage.

[pdmack]

Couldn't get this to build (based on updated CONTRIBUTING.md)

#27 7.962 tar: Python-3.8.12/Include/*: Not found in archive
#27 7.962 tar: Exiting with failure status due to previous errors
#27 9.241 tar: Python-3.8.12/Misc/*: Not found in archive
#27 9.241 tar: Exiting with failure status due to previous errors
#27 10.52 tar: Python-3.8.12/Modules/*: Not found in archive
#27 10.52 tar: Exiting with failure status due to previous errors
#27 11.80 tar: Python-3.8.12/Objects/*: Not found in archive
#27 11.80 tar: Exiting with failure status due to previous errors
#27 13.05 tar: Python-3.8.12/Python/*: Not found in archive
#27 13.05 tar: Exiting with failure status due to previous errors
#27 ERROR: executor failed running [/bin/bash -c if [[ "${MORPHEUS_WITH_PYDEBUG}" =~ ^(TRUE|True|true|1)$ ]]; then       source activate morpheus       && export source_file=$( ls /opt/conda/conda-bld/src_cache/Python-${PYTHON_VER}*.tar.xz )       && export install_file=$( ls /opt/conda/conda-bld/linux-64/python-${PYTHON_VER}*.tar.bz2 )       && mamba install --use-local ${install_file}       && export debug_py_ver=$(python --version | cut -d ' ' -f2)       && cp -R /opt/conda/envs/morpheus/include/python${PYTHON_VER}d/*         /opt/conda/envs/morpheus/include/python${PYTHON_VER}       && for src in Include Misc Modules Objects Python; do            tar --strip-components=1 --extract --wildcards --file=${source_file} Python-${debug_py_ver}/${src}/*           && mkdir -p /usr/local/src/python-dbg/${src}           && cp -R ./${src} /usr/local/src/python-dbg/${src}           ;          done       && cp Misc/gdbinit $(eval echo "~${MORPHEUS_USER}")/.gdbinit       ;     fi]: exit code: 2
------
 > [development 3/3] RUN --mount=type=bind,from=development_deps,source=/opt/conda/conda-bld,target=/opt/conda/conda-bld     if [[ "true" =~ ^(TRUE|True|true|1)$ ]]; then       source activate morpheus       && export source_file=$( ls /opt/conda/conda-bld/src_cache/Python-3.8*.tar.xz )       && export install_file=$( ls /opt/conda/conda-bld/linux-64/python-3.8*.tar.bz2 )       && mamba install --use-local ${install_file}       && export debug_py_ver=$(python --version | cut -d ' ' -f2)       && cp -R /opt/conda/envs/morpheus/include/python3.8d/*         /opt/conda/envs/morpheus/include/python3.8       && for src in Include Misc Modules Objects Python; do            tar --strip-components=1 --extract --wildcards --file=${source_file} Python-${debug_py_ver}/${src}/*           && mkdir -p /usr/local/src/python-dbg/${src}           && cp -R ./${src} /usr/local/src/python-dbg/${src}           ;          done       && cp Misc/gdbinit $(eval echo "~root")/.gdbinit       ;     fi:
#27 7.962 tar: Python-3.8.12/Include/*: Not found in archive
#27 7.962 tar: Exiting with failure status due to previous errors
#27 9.241 tar: Python-3.8.12/Misc/*: Not found in archive
#27 9.241 tar: Exiting with failure status due to previous errors
#27 10.52 tar: Python-3.8.12/Modules/*: Not found in archive
#27 10.52 tar: Exiting with failure status due to previous errors
#27 11.80 tar: Python-3.8.12/Objects/*: Not found in archive
#27 11.80 tar: Exiting with failure status due to previous errors
#27 13.05 tar: Python-3.8.12/Python/*: Not found in archive
#27 13.05 tar: Exiting with failure status due to previous errors
------
error: failed to solve: executor failed running [/bin/bash -c if [[ "${MORPHEUS_WITH_PYDEBUG}" =~ ^(TRUE|True|true|1)$ ]]; then       source activate morpheus       && export source_file=$( ls /opt/conda/conda-bld/src_cache/Python-${PYTHON_VER}*.tar.xz )       && export install_file=$( ls /opt/conda/conda-bld/linux-64/python-${PYTHON_VER}*.tar.bz2 )       && mamba install --use-local ${install_file}       && export debug_py_ver=$(python --version | cut -d ' ' -f2)       && cp -R /opt/conda/envs/morpheus/include/python${PYTHON_VER}d/*         /opt/conda/envs/morpheus/include/python${PYTHON_VER}       && for src in Include Misc Modules Objects Python; do            tar --strip-components=1 --extract --wildcards --file=${source_file} Python-${debug_py_ver}/${src}/*           && mkdir -p /usr/local/src/python-dbg/${src}           && cp -R ./${src} /usr/local/src/python-dbg/${src}           ;          done       && cp Misc/gdbinit $(eval echo "~${MORPHEUS_USER}")/.gdbinit       ;     fi]: exit code: 2

[drobison00]

@pdmack That's odd, for some reason the install script isn't running in the morpheus conda environment for you, so its detecting the python version in the base environment... I'll investigate more. Not at all clear why it works on my side.

Edit: I'd switched to using the wrong environment variables in the build file. Conda build defines PY_VER, but I was using PYTHON_VER, which broke some post build symbolic links. Fixed now.

[drobison00]

@pdmack @mdemoret-nv All the updates are in; morpheus builds without any issues in container, tests passing, etc.. should be good to go if there are no other concerns.

[pdmack]

LGTM
Not sure if we want to sprinkle in a pdb breakpoint example or some such?

[pdmack]

LGTM

[drobison00]

LGTM Not sure if we want to sprinkle in a pdb breakpoint example or some such?

If @mdemoret-nv doesn't have any other change requests, I'll add a quick example of breaking within a the cpython evaluation loop and printing object contents.

[mdemoret-nv]

Looks good. Only one small question.

[mdemoret-nv]

@gpucibot merge