Always use regexp-based wildignore handling

In 7489118 I added this for the watchman scanner. Here I make all scanners
behave the same way. Rationale: I want to remove as many `VIM` calls from the
finder/scanner classes as possible, so that I can use them as-is in the
`commandtd` process. Bonus: simplicity of maintaining only one code path,
probably faster. Risk: possible behavior change, depending on config. Fucks
given: NaN (it's complicated).

doc/command-t.txt

@@ -1126,25 +1126,6 @@ settings can be used to control behavior:
       |g:CommandTWildIgnore| setting to apply an override that takes effect
       only during Command-T searches.
 
-      Note that there are some differences among file scanners
-      (see |g:CommandTFileScanner|) with respect to 'wildignore' handling:
-
-      - The default "ruby" scanner explores the filesystem recursively using a
-        depth-first search, and any directory (or subdirectory) which matches
-        the 'wildignore' pattern is not explored. So, if your 'wildignore'
-        contains "*/node_modules" then that entire sub-hierarchy will be
-        ignored.
-
-      - The "git" and "find" scanners apply 'wildignore' filtering only after
-        completing their scans.
-
-      - The "watchman" scanner is intended for use with massive hierarchies
-        where speed is of the utmost import, so it doesn't make use of Vim's
-        very slow 'wildignore' filtering at all. Instead, it constructs a
-        regular expression on the fly based on 'wildignore' and/or
-        |g:CommandTWildIgnore| and uses that. To suppress this behavior, set
-        |g:CommandTWildIgnore| to an empty string.
-
 
 INTEGRATION AND EXTENSIBILITY         *command-t-integration-and-extensibility*
 
@@ -1318,9 +1299,7 @@ On a large hierarchy with of the order of 500,000 files, scanning a directory
 tree with a tool like the `find` executable may take literally minutes with a
 cold cache. Once the cache is warm, the same `find` run may take only a second
 or two. Command-T provides a "find" scanner to leverage this performance, but
-there is still massive overhead in passing the results through Vim internal
-functions that apply 'wildignore' settings and such, so for truly immense
-repos the "watchman" scanner is the tool of choice.
+for truly immense repos the "watchman" scanner is the tool of choice.
 
 This scanner delegates the task of finding files to Facebook's `watchman` tool
 (https://github.com/facebook/watchman), which can return results for a 500,000
@@ -1552,6 +1531,9 @@ HISTORY                                         *command-t-history*
 next (not yet released)
 
 - Command-T now uses |:map-<nowait>|, when available, when setting up mappings.
+- 'wildignore' filtering is now always performed by producing an equivalent
+  regular expression and applying that (previously this only happened with
+  the "watchman" file scanner).
 
 4.0 (16 May 2016)
 

ruby/command-t/lib/command-t/controller.rb

@@ -545,6 +545,14 @@ def mru_finder
       @mru_finder ||= CommandT::Finder::MRUBufferFinder.new
     end
 
+    def wildignore
+      ignore = VIM::get_string('g:CommandTWildIgnore')
+      if ignore.nil? && VIM::exists?('&wildignore')
+        ignore = ::VIM::evaluate('&wildignore').to_s
+      end
+      VIM::wildignore_to_regexp(ignore) unless ignore.nil?
+    end
+
     def file_finder
       @file_finder ||= CommandT::Finder::FileFinder.new nil,
         :max_depth              => VIM::get_number('g:CommandTMaxDepth'),
@@ -553,7 +561,7 @@ def file_finder
         :always_show_dot_files  => VIM::get_bool('g:CommandTAlwaysShowDotFiles'),
         :never_show_dot_files   => VIM::get_bool('g:CommandTNeverShowDotFiles'),
         :scan_dot_directories   => VIM::get_bool('g:CommandTScanDotDirectories'),
-        :wild_ignore            => VIM::get_string('g:CommandTWildIgnore'),
+        :wildignore             => wildignore,
         :scanner                => VIM::get_string('g:CommandTFileScanner'),
         :git_scan_submodules    => VIM::get_bool('g:CommandTGitScanSubmodules')
     end

ruby/command-t/lib/command-t/scanner/file_scanner.rb

@@ -25,16 +25,15 @@ def initialize(path = Dir.pwd, options = {})
         @max_files            = options[:max_files] || 100_000
         @max_caches           = options[:max_caches] || 1
         @scan_dot_directories = options[:scan_dot_directories] || false
-        @wild_ignore          = options[:wild_ignore]
+        @wildignore           = options[:wildignore]
         @scan_submodules      = options[:git_scan_submodules] || false
-        @base_wild_ignore     = wild_ignore
       end
 
       def paths
         @paths[@path] ||= begin
           ensure_cache_under_limit
           @prefix_len = @path.chomp('/').length + 1
-          set_wild_ignore { paths! }
+          paths!
         end
       end
 
@@ -64,10 +63,6 @@ def show_max_files_warning
         end
       end
 
-      def wild_ignore
-        VIM::exists?('&wildignore') && ::VIM::evaluate('&wildignore').to_s
-      end
-
       def paths!
         raise RuntimeError, 'Subclass responsibility'
       end
@@ -82,34 +77,13 @@ def ensure_cache_under_limit
       end
 
       def path_excluded?(path, prefix_len = @prefix_len)
-        if apply_wild_ignore?
-          # first strip common prefix (@path) from path to match VIM's behavior
+        if @wildignore
+          # First strip common prefix (@path) from path to match VIM's behavior.
           path = path[prefix_len..-1]
           path = VIM::escape_for_single_quotes path
-          ::VIM::evaluate("empty(expand(fnameescape('#{path}')))").to_i == 1
+          path =~ @wildignore
         end
       end
-
-      def has_custom_wild_ignore?
-        !!@wild_ignore
-      end
-
-      # Used to skip expensive calls to `expand()` when there is no applicable
-      # wildignore.
-      def apply_wild_ignore?
-        if has_custom_wild_ignore?
-          !@wild_ignore.empty?
-        else
-          !!@base_wild_ignore
-        end
-      end
-
-      def set_wild_ignore(&block)
-        ::VIM::command("set wildignore=#{@wild_ignore}") if has_custom_wild_ignore?
-        yield
-      ensure
-        ::VIM::command("set wildignore=#{@base_wild_ignore}") if has_custom_wild_ignore?
-      end
     end
   end
 end

ruby/command-t/lib/command-t/scanner/file_scanner/watchman_file_scanner.rb

@@ -43,11 +43,8 @@ def paths!
 
             # could return error if watch is removed
             extracted = extract_value(paths, 'files')
-            if (
-              apply_wild_ignore? &&
-              (regex = VIM::wildignore_to_regexp(@wild_ignore || @base_wild_ignore))
-            )
-              extracted.select { |path| path !~ regex }
+            if @wildignore
+              extracted.select { |path| path !~ @wildignore }
             else
               extracted
             end

spec/command-t/finder/buffer_finder_spec.rb

@@ -37,9 +37,8 @@
       expect(@finder.sorted_matches_for('i')).to include('.vim/notes')
     end
 
-    it "does not use the Vim expand() function to consult the 'wildignore' setting" do
-      do_not_allow(::VIM).evaluate
-      @finder.sorted_matches_for('i')
+    it "does not consult the 'wildignore' setting" do
+      expect(@finder.sorted_matches_for('').count).to eq(5)
     end
 
     it 'obeys the :limit option for empty search strings' do

spec/command-t/scanner/file_scanner/ruby_file_scanner_spec.rb

@@ -13,7 +13,6 @@
 
     stub(::VIM).evaluate(/exists/) { 1 }
     stub(::VIM).evaluate(/expand\(.+\)/) { '0' }
-    stub(::VIM).evaluate(/wildignore/) { '' }
     stub(::VIM).command(/echon/)
     stub(::VIM).command('redraw')
   end
@@ -40,20 +39,18 @@
 
   describe "'wildignore' exclusion" do
     context "when there is a 'wildignore' setting in effect" do
-      it "calls on VIM's expand() function for pattern filtering" do
-        stub(::VIM).command(/set wildignore/)
+      it "filters out matching files" do
         scanner =
-          CommandT::Scanner::FileScanner::RubyFileScanner.new @dir, :wild_ignore => '*.o'
-        mock(::VIM).evaluate(/expand\(.+\)/).times(10)
-        scanner.paths
+          CommandT::Scanner::FileScanner::RubyFileScanner.new @dir,
+            :wildignore => CommandT::VIM::wildignore_to_regexp('xyz')
+        expect(scanner.paths.count).to eq(@all_fixtures.count - 1)
       end
     end
 
     context "when there is no 'wildignore' setting in effect" do
-      it "does not call VIM's expand() function" do
+      it "does nothing" do
         scanner = CommandT::Scanner::FileScanner::RubyFileScanner.new @dir
-        mock(::VIM).evaluate(/expand\(.+\)/).never
-        scanner.paths
+        expect(scanner.paths.count).to eq(@all_fixtures.count)
       end
     end
   end