.submodules leads to IndexError: list index out of range on info = self._cache[item]

[yarikoptic]

not sure if related to still open #279

I have bunch of freshly added, and then also modified submodules. That leads to following kaboom:

$> pip install --upgrade GitPython
Collecting GitPython
  Downloading GitPython-2.1.3-py2.py3-none-any.whl (442kB)
Requirement already up-to-date: gitdb2>=2.0.0 in /home/yoh/proj/datalad/datalad/venv-tests/lib/python2.7/site-packages (from GitPython)
Requirement already up-to-date: smmap2>=2.0.0 in /home/yoh/proj/datalad/datalad/venv-tests/lib/python2.7/site-packages (from gitdb2>=2.0.0->GitPython)
Installing collected packages: GitPython
  Found existing installation: GitPython 2.1.0
    Uninstalling GitPython-2.1.0:
      Successfully uninstalled GitPython-2.1.0
Successfully installed GitPython-2.1.3
You are using pip version 7.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

$> python -c 'from git import Repo; print Repo(".").submodules'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/yoh/proj/datalad/datalad/venv-tests/local/lib/python2.7/site-packages/git/repo/base.py", line 290, in submodules
    return Submodule.list_items(self)
  File "/home/yoh/proj/datalad/datalad/venv-tests/local/lib/python2.7/site-packages/git/util.py", line 932, in list_items
    out_list.extend(cls.iter_items(repo, *args, **kwargs))
  File "/home/yoh/proj/datalad/datalad/venv-tests/local/lib/python2.7/site-packages/git/objects/submodule/base.py", line 1181, in iter_items
    sm = rt[p]
  File "/home/yoh/proj/datalad/datalad/venv-tests/local/lib/python2.7/site-packages/git/objects/tree.py", line 293, in __getitem__
    info = self._cache[item]
IndexError: list index out of range

$> git submodule
+60d265e09441a6316f7c3336c0eba4370afed2ac 2009 (heads/master)
+5cdff4402dec87be62ae0ba197c63ad1da7ba2c2 2010 (heads/master)
+e5638100b6e4514a6e6e7379a581fb2801b7a59b 2011 (heads/master)
+62bf8ad207a8d70706a3b1288e2aaa6b1efe20ed 2013 (heads/master)
+214a854c94d9d8c5283643a59fb633f46563fbd0 2014 (heads/master)
+a6770f5aa5c8552b3cd803b138e5ae71c281188b 2015 (heads/master)
+4790ba30c6e666c86ec26d78042eb2a82cb043f8 2016 (heads/master)
+bf191a0120bce11be84f3e9a260eb830939fecbc 2017 (heads/master)

$> git diff
diff --git a/2009 b/2009
index b547198..60d265e 160000
--- a/2009
+++ b/2009
@@ -1 +1 @@
-Subproject commit b547198f5c417a88fa3e683c7a73b2ba014b2331
+Subproject commit 60d265e09441a6316f7c3336c0eba4370afed2ac
...

$> head .gitmodules 
[submodule "2009"]
	path = 2009
	url = ./2009
[submodule "2010"]
	path = 2010
	url = ./2010
[submodule "2011"]
	path = 2011
	url = ./2011

Failed to replicate with a simplistic

$> rm -rf /tmp/test && mkdir /tmp/test && cd /tmp/test; git init; echo smth > smth; git add smth; git commit -m smth; mkdir sub; cd sub; git init; echo 123 > 123; git add 123; git commit -m added ; cd ../; git submodule add ./sub sub; cd sub; echo 124 >| 123; git add 123; git commit -m changed; cd ..; git submodule; python -c 'from git import Repo; print Repo(".").submodules'
Initialized empty Git repository in /tmp/test/.git/
[master (root-commit) bbfc6e1] smth
 1 file changed, 1 insertion(+)
 create mode 100644 smth
Initialized empty Git repository in /tmp/test/sub/.git/
[master (root-commit) 6063d92] added
 1 file changed, 1 insertion(+)
 create mode 100644 123
smth  sub/
Adding existing repo at 'sub' to the index
123
[master 4a3e397] changed
 1 file changed, 1 insertion(+), 1 deletion(-)
smth  sub/
+4a3e397542e87ff842555f8f6c6fdfafffa0c375 sub (heads/master)
[git.Submodule(name=sub, path=sub, url=./sub, branch_path=refs/heads/master)]
changes on filesystem:                                                                                                                                                                                                                       
 sub | 2 +-
cached/staged changes:
 .gitmodules | 3 +++
 sub         | 1 +

so I guess I have missed some aspect

[Byron]

Thanks for letting me know. I assume that the original repository that is showing the issue is not public, so we are left with an issue that is not reproducible.
What do you suggest we should do?

[yarikoptic]

eh -- took me a while to find that repo where I originally ran into it (again blaming myself for not quoting enough info for myself)... found it -- now you can get the entirety of it in its dirty state from http://www.onerussian.com/tmp/phsyionet-challenge.tar.gz

[Byron]

@yarikoptic Thanks for taking the time to look - I pulled the file and was able to easily reproduce the issue. Just to be on the safe side, I put it into a gist.

This is how to reproduce the issue, assuming you have gitpython in your PYTHON_PATH:

set -e
curl https://gist.githubusercontent.com/Byron/17073cdb4c3019675d34a4705339e962/raw/802129991aa0bd2c23ee0f600b2ec4cef88ffbad/challenge.base64.zip | base64 -D > challenge.zip
unzip challenge.zip
(cd challenge && python -c 'from git import Repo; print Repo(".").submodules')

[yarikoptic]

In datalad/datalad#1801 we ran into our datalad crashing because we used GitPython's parser for .gitmodules and used get_value which automagically casts values into int and bool whenever it could. We fixed it on our end, but then decided to test more... Now I have managed to reproduce both crashes -- the one we have in that datalad issue (when I use bool value for the name) and this one (when I use numeric), so I guess the reason could be the same "misuse" of get_value (casts) whenever get should have been used (no casting):

clean run:

$> s=s123; builtin cd /tmp/; rm -rf /tmp/gitsub; mkdir /tmp/gitsub; cd /tmp/gitsub; git init; mkdir $s; cd $s; git init; touch 1; git add 1; git commit -m msg 1; cd ..; git submodule add ./$s $s; git add .gitmodules; git commit -m "added subm $s" 
Initialized empty Git repository in /tmp/gitsub/.git/
Initialized empty Git repository in /tmp/gitsub/s123/.git/
[master (root-commit) 2b57b47] msg
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 1
s123/
Adding existing repo at 's123' to the index
[master (root-commit) 8990a73] added subm s123
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 s123
2 10278.....................................:Thu 07 Sep 2017 12:11:07 PM EDT:.
(git)hopa:/tmp/gitsub[master]git
$> python -c 'import git; r=git.Repo("."); print(r.submodules)'
[git.Submodule(name=s123, path=s123, url=./s123, branch_path=refs/heads/master)]

int:

2 10279.....................................:Thu 07 Sep 2017 12:11:10 PM EDT:.
(git)hopa:/tmp/gitsub[master]git
$> s=123; builtin cd /tmp/; rm -rf /tmp/gitsub; mkdir /tmp/gitsub; cd /tmp/gitsub; git init; mkdir $s; cd $s; git init; touch 1; git add 1; git commit -m msg 1; cd ..; git submodule add ./$s $s; git add .gitmodules; git commit -m "added subm $s" 
Initialized empty Git repository in /tmp/gitsub/.git/
Initialized empty Git repository in /tmp/gitsub/123/.git/
[master (root-commit) 8297bb7] msg
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 1
123/
Adding existing repo at '123' to the index
[master (root-commit) 3417128] added subm 123
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 123
2 10280.....................................:Thu 07 Sep 2017 12:11:13 PM EDT:.
(git)hopa:/tmp/gitsub[master]git
$> python -c 'import git; r=git.Repo("."); print(r.submodules)'
Traceback (most recent call last):                             
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/git/repo/base.py", line 293, in submodules
    return Submodule.list_items(self)
  File "/usr/lib/python2.7/dist-packages/git/util.py", line 932, in list_items
    out_list.extend(cls.iter_items(repo, *args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/git/objects/submodule/base.py", line 1181, in iter_items
    sm = rt[p]
  File "/usr/lib/python2.7/dist-packages/git/objects/tree.py", line 293, in __getitem__
    info = self._cache[item]
IndexError: list index out of range

bool:

$> s=false; builtin cd /tmp/; rm -rf /tmp/gitsub; mkdir /tmp/gitsub; cd /tmp/gitsub; git init; mkdir $s; cd $s; git init; touch 1; git add 1; git commit -m msg 1; cd ..; git submodule add ./$s $s; git add .gitmodules; git commit -m "added subm $s"   
Initialized empty Git repository in /tmp/gitsub/.git/
Initialized empty Git repository in /tmp/gitsub/false/.git/
[master (root-commit) 987d30b] msg
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 1
false/
Adding existing repo at 'false' to the index
[master (root-commit) 12afd38] added subm false
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 false
2 10282.....................................:Thu 07 Sep 2017 12:14:50 PM EDT:.
(git)hopa:/tmp/gitsub[master]git
$> python -c 'import git; r=git.Repo("."); print(r.submodules)'
Traceback (most recent call last):                             
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/git/repo/base.py", line 293, in submodules
    return Submodule.list_items(self)
  File "/usr/lib/python2.7/dist-packages/git/util.py", line 932, in list_items
    out_list.extend(cls.iter_items(repo, *args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/git/objects/submodule/base.py", line 1194, in iter_items
    sm._name = n
AttributeError: 'Blob' object has no attribute '_name'