Guard confused by symlinked folders

[ITler]

Hope filing the issue here is correct.
When setting up my dev environment (current Ubuntu 14LTS, XFS filesystem, current rvm with ruby 2.2.1 and current gems) for coding puppet modules I ran into problems indicated by duplicate directories error.
That's why I follow puppet module development recommendations. When I get to the point of using rspec, I've got following puppet module structure (tool based creation):

my_module /                            
- manifests                   
  - init.pp                   
- spec                        
  - classes                   
    - init_spec.rb            
  - fixtures                  
    - modules                 
      - my_module ---> symlink to module root
[...]

When starting guard and the directory symlink spec/fixtures/modules/my_module already exists then guard (listen gem) is not able to recognize file modifications within my entire module path.
On guard pry console Guard.state.session.watchdirs shows correct path.

#Guardfile
ignore %r{spec/fixtures} # does not help
notification :libnotify

guard 'rspec', :cmd => "bundle exec rspec", :cmd_additional_args => '-f d -c --exclude-pattern "spec/fixtures/**/*"' do # --exclude-pattern does not help
  watch(%r{^spec/.+_spec\.rb$})
  watch('spec/spec_helper.rb')  { "spec" }
end

guard 'rspec', :cmd => "bundle exec rspec", :cmd_additional_args => '-f d -c --fail-fast --exclude-pattern "spec/fixtures/**/*"' do # --exclude-pattern does not help
  watch(%r{^manifests/(.+)\.pp$})  { |m| "spec/classes/#{m[1]}_spec.rb" }
end

When creating spec/fixtures/modules/my_module after starting guard (usually using rake spec_prep), guard (listen) is sometimes working as expected. Sometimes guard doesn't recognize changes to manifest files (manifests/*.pp), sometimes changes to manifest's spec files (spec/*_spec.rb). So this work-around is not stable.
I've just found your Understanding-Guard guide. Great, I've found out, that using -p option will allow guard to work properly even if symlinked spec/fixtures/modules/my_module exists before starting guard. And this seems to be stable behaviour.

I couldn't work out answers to my follow up questions. Maybe, you could answers them. I would like to know:

1. What are the detriments of using -p? There should be any (like increased CPU usage?), otherwise polling would be default behaviour of guard.
2. Will there be an update of guard (or listen) that solidly solves that issue?
3. What did I miss, in general?
4. If this won't / can't be fixed shortly, I would need a stable and easy to handle work-around. What else could I do without changing the overall module file structure? (This seems to effect all users using this workflow)

[e2]

At this level you're quickly getting into complexities that are normally hidden from average users.

Listen tries to be "portable" and "easy", but there are severe filemonitoring implementation limitations. Very severe. That's just to let you know you're entering the deep end.

First, you've created a "filesystem loop". Any tool which recurses and follows symlinks will loop infinitely. The UNIX tool find will likely fail with an error on your setup.

Listen should detect this and fail with an error - which means it simply won't respond to changes.

I didn't find a reference in the Puppet tutorial you mentioned that says to make such a symlink - I'd avoid it anyway.

Second, Listen strongly discourages physically watching the same directory more than once - this doesn't make sense and it's a waste of resources.

It's convenient to watch "a whole project", but you usually only want to watch a few specific directories. If you want a specific layout, I'd suggest moving the physical directories into a special directory (e.g. src), watch only that directory, and setup the project structure using symlinks pointing to the real directories inside src (or whatever).

Symlink support is extremely difficult to implement in a way that makes people happy - one reason is that the OSX implementation forces recursion anyway, and Listen can't be portable if this doesn't apply to all adapters.

My suggestions are:

1. remove the filesystem loop anyway (because other tools, libraries and editors may choke anyway)
2. avoid watching directories with symlinks - watch only selected physical directories (remember that recursion is forced on OSX, and so sadly this limitation infects other adapters)

In short - make heavy use of symlinks to defining a structure to match the workflow, while completely avoiding symlinks in watched directories.

File-monitoring and symlinks just will never work "as expected". You can pretty much blame Apple for that.

Let me know if that helps.

[ITler]

Thanks for entering the deep end ;)
Sorry for being imprecise regarding puppet tutorial reference... The linked guide tells to use puppet module generate to create an appropriate module structure. For testing the usage of rspec-puppet is recommended. Following setup instructions will lead to test skeleton creation (using rspec-puppet-init). This will create those symlinks to set up testing environment appropriately.

1. I've understood why to avoid those fs loops.
2. I could get used to use special physical directories as you advised, but this will lead to more complexity at file level and maybe (if possible) customization overhead to other tools in the stack. Indeed, it would be simpler if one could tell Listen (via Guardfile) to really ignore files and directories. Then, using ignore %r{spec/fixtures/} would lead to a minimum and more transparent configuration (rather than directories %w(manifests files config spec/classes spec/defines spec/functions spec/and-so-on)) and no dependent changes to the file structure for avoiding fs loops. Of cause, I don't see all use cases and technical implications. Maybe it is technically not possible to realise this.

Again thank you.

[e2]

I could get used to use special physical directories as you advised, but this will lead to more complexity at file level and maybe (if possible) customization overhead to other tools in the stack.

With symlinks you'll have exactly the same layout - with the exception of an additional folder with the "originals". Unlike other tools, Listen needs to distinguish between physical dirs and symlinks because file monitoring happens "below" the filesystem (where the differences between folders and symlinks are taken into account).

Indeed, it would be simpler if one could tell Listen (via Guardfile) to really ignore files and directories.

Again, the OSX adapter is recursive - and so there's no way to "physically" ignore a subtree. Basically, watching the project root dir means watching the whole project tree. And since the OSX driver is recursive, it doesn't make sense for recursion to be configurable (e.g. on Linux).

Also, since you can create or rename an ignored directory at any time, handling all the edge cases becomes extremely complex quickly (adding/removing folders from being watched, etc.) - reorganizing a project layout is a much better investment hands down, every time.

Also, :ignore is a "post-filter", which is applied to events already received. What you'd need for your case is a "pre-filter", which decides what to watch or not.

For a simpler configuration, you can probably use something like

directories Dir['spec/*'].reject { |d| d =~ /^spec\/fixtures\// } # + (...)

It maybe wouldn't be hard to implement Linux-only configuration options and intelligently avoid watching already-watched directories. I wouldn't mind accepting PRs for this - my own todo list is already a mile long though for me to tackle this myself.

I still think the need for symlinks can be replaced with better project configuration (there shouldn't be a need to symlink the whole project within fixtures at all).

[ITler]

With symlinks you'll have exactly the same layout - [...]

Yeah, I've already reorganised some of my projects and I start liking it as I can put Guardfile, Rakefile and so on into a separate repository for standardization. (It seems they won't change for puppet module development, individually.)

Again, the OSX adapter is recursive -

OK, now I got it.

At the end, again, I'd like to thank you for taking time to deep-diving the topic here.