Projectile won't index some repos

[trishume]

After upgrading to v0.2.0 I'm a problem with projectile where projectile won't find the files in some repos. I think the pattern might be repos that I've visited while my emacs was in a borked state, so it might have something to do with caching.

On some repos when I do either a find in project or a helm find, the only files that show up are the ones I've opened in emacs. When I search for any other file, even the exact names of files I know exist, nothing comes up.

This only seems to happen on repos I've visited in emacs within the last little while, i'm not sure what the exact time period is. If I open a project I haven't touched in a long time projectile works fine.

I'll keep looking into this, I think it has something to do with the caching.

[syl20bnr]

Do you have a .projectile file in those repo ?
Normally the .git file should enable caching it was not the case when I tested, I had to create a .projectile.

[trishume]

Interesting, deleting the cache and bookmarks fixes it in one project that was previously broken but others don't seem to work.

And no I don't have .projectile files in those repos, but under no circumstances will I be satisfied with adding those files being a solution, I have too many projects to bother with projectile files. The repos where it works all don't have projectile files, in fact the only projectile file on my system is the spacemacs one.

[syl20bnr]

Understandable :-)
I will dig more into projectile sources to understand why it's happening.

[trishume]

Aha, I figured out something: the repos it breaks for are anything I open after using projectile in the spacemacs repo.

Steps To Reproduce (at least for me)

1. Delete projectile cache and bookmarks
2. Open emacs
3. Try projectile on a bunch of git repos (works for all of them!)
4. Open spacemacs project, run projectile (fails)
5. Open any git repo, even ones that had previously worked on past run-throughs, run projectile (fails!)
6. Edit added steps: close emacs and reopen
7. Visit one of the repos that had been caused to fail on previous run, run projectile (fails!)

Edit: By fails I mean doesn't find any files, and by run projectile I mean <SPC> p h

[syl20bnr]

Nice find ! So the first different thing I see is the .projectile file in the spacemacs repo.

[trishume]

Yah, that's probably the case. I remember before the changes projectile was broken in spacemacs too (that hash-table-p) thing, but it didn't affect anything else. The contamination has spread!

It also seems that the brokenness is persisted in the cache, because even if I restart emacs and open those projects again (without opening spacemacs) they are broken. Until I delete the cache file, then everything works until I open spacemacs.

[syl20bnr]

Something weird here, why projectile does not create a different cache file per project ? There must be something wrong in the spacemacs configuration.

[trishume]

It's supposed to create multiple cache files? Where is it supposed to put them, in the project directory?

I would prefer that it kept them all in one place, I don't want to have to update the gitignore of every project I edit to ignore some cache file.

[trishume]

Can you reproduce this? If you can't I'm happy to follow debugging leads for you if you have any ideas on where to look.

[syl20bnr]

I was assuming that projectile creates one cache per project, which would make sense with the bookmarks files (to locate all the cache files).
But it does not seem to be the case.

I would prefer one cache per project because it provides better isolation and thus it better scales. Right now, all the projects are artificially bound together via this cache which may be the root cause of the issue you discovered here, maybe not.

I don't mind adding an entry in .gitignore since this is always something you have to do when you bootstrap a project (moreover 5 minutes of elisp script and you're done with this concern ;-)).

[trishume]

Ok I did a bunch of testing and I now understand the actual behaviour, my previous theories were wrong. There is nothing special about the spacemacs repo.

Only the first project loaded after emacs is opened gets properly cached. Any subsequently projects that are opened have their file list cached as the first file you visit. If you restart emacs and open a project that is in the cache but only has one file registered, it will only have that one file, but if the first project you open is not in the cache it will index it properly.

So probably what is happening is that something in projectile is adding only the one file to the cache, and then not indexing because it has already cached things.

[trishume]

Understandably, disabling caching fixes this behaviour, but that's just a temporary fix so that I can use my most-used spacemacs command (find in project) until we figure this out.

[syl20bnr]

There is a function to invalidate the cache in projectile, did you experiment with it ? It should be possible to do it per project.

Normally with the universal argument, some projectile commander commands invalidates first the cache. But the universal argument on <SPC> u in spacemacs does nothing in this case :-(

Note: about my comment before, having to add .projectile to every git repo is indeed a no go. I don't want to "pollute" the repo I fork. So I'm with you on this one :-)
The perfect solution would be a mix of the 2 approaches.

[trishume]

Agreed, I was thinking ideally projectile.cache would be a folder with a bunch of files that are the caches for each project.

The problem with the current method is that if you have one project that has a TON of subfiles then that cache will be loaded up even if you aren't using it.

[punassuming]

I had an issue with this for a while. In my case, it was the submodule command used in Projectile, which was not compatible with Windows.

Fix referenced here:
bbatsov/projectile#376

[tuhdo]

This issue is fixed: bbatsov/projectile#519

You can reproduce by the steps I listed there. Now you can enable caching again for fast file list startup and never have the issue that only a few opened files are cached. You should enable caching, since it makes Projectile start instantly even with many ten thousands of files.

The submodule issue is a different problem though.

PS: You can close this issue now :)

[trishume]

Yay thanks @tuhdo you're awesome!

[syl20bnr]

@tuhdo 👍