Auto merge of #818 - micbou:filter-and-sort-candidates-benchmark, r=b…

…staletic [READY] Add FilterAndSortCandidates benchmarks [The `CandidatesForQuery` method](https://github.com/Valloric/ycmd/blob/2575bdb8c83dd5ace3d6f3d72a3425fdf2c18f5e/cpp/ycm/IdentifierCompleter.h#L69) is not the only place where performance is critical. There is also [the `FilterAndSortCandidates` function](https://github.com/Valloric/ycmd/blob/2575bdb8c83dd5ace3d6f3d72a3425fdf2c18f5e/cpp/ycm/PythonSupport.h#L29) which is used in the Python layer to filter and sort candidates returned by completers other than the identifier one (while `CandidatesForQuery` is specific to the identifier completer). We should add benchmarks for this function too. It would be great for these benchmarks to be based on real usage (e.g. case 1 or 2 from ycm-core/YouCompleteMe#2668). This could be done by storing the candidates in a file that would be read by the benchmarks to get the candidates. However, this kind of file would be too big (> 1MB) to be committed to the repository. We would have to download the file from somewhere else to run the benchmarks. I didn't want to bother so I went for the same candidates as the `CandidatesForQuery` benchmark. Those candidates represent a corner case but, as long as we don't specifically target them, they should be good enough. Two benchmarks are added: one when the candidates are not already stored in the repository and another when they are. This is important because both situations arise in practice: when filtering and sorting the candidates for the first time, they are added to the repository (user pressing `<C-Space>`) then they are retrieved from the repository next times (user typing characters to filter out candidates). Here are the benchmark results on my configuration: ``` Run on (4 X 3504 MHz CPU s) ------------------------------------------------------------------------------------------------------------------ Benchmark Time CPU Iterations ------------------------------------------------------------------------------------------------------------------ IdentifierCompleterFixture/CandidatesWithCommonPrefix/1024 512467 ns 514442 ns 1122 IdentifierCompleterFixture/CandidatesWithCommonPrefix/2048 1144025 ns 1143845 ns 641 IdentifierCompleterFixture/CandidatesWithCommonPrefix/4096 2643405 ns 2659108 ns 264 IdentifierCompleterFixture/CandidatesWithCommonPrefix/8192 6263122 ns 6267897 ns 112 IdentifierCompleterFixture/CandidatesWithCommonPrefix/16384 13002049 ns 12535795 ns 56 IdentifierCompleterFixture/CandidatesWithCommonPrefix/32768 28741026 ns 28704184 ns 25 IdentifierCompleterFixture/CandidatesWithCommonPrefix/65536 60231116 ns 60840390 ns 10 IdentifierCompleterFixture_BigO 57.59 NlgN 57.96 NlgN IdentifierCompleterFixture_RMS 1 % 2 % PythonSupportFixture/FilterAndSortUnstoredCandidatesWithCommonPrefix/1024 4280306 ns 4680030 ns 150 PythonSupportFixture/FilterAndSortUnstoredCandidatesWithCommonPrefix/2048 9249671 ns 9186726 ns 90 PythonSupportFixture/FilterAndSortUnstoredCandidatesWithCommonPrefix/4096 18832170 ns 18373451 ns 45 PythonSupportFixture/FilterAndSortUnstoredCandidatesWithCommonPrefix/8192 38904981 ns 37266906 ns 18 PythonSupportFixture/FilterAndSortUnstoredCandidatesWithCommonPrefix/16384 78318612 ns 78000500 ns 9 PythonSupportFixture/FilterAndSortUnstoredCandidatesWithCommonPrefix/32768 158404771 ns 163801050 ns 4 PythonSupportFixture/FilterAndSortUnstoredCandidatesWithCommonPrefix/65536 317453915 ns 319802050 ns 2 PythonSupportFixture_BigO 4836.93 N 4891.16 N PythonSupportFixture_RMS 1 % 2 % PythonSupportFixture/FilterAndSortStoredCandidatesWithCommonPrefix/1024 542299 ns 530403 ns 1000 PythonSupportFixture/FilterAndSortStoredCandidatesWithCommonPrefix/2048 1264767 ns 1279153 ns 561 PythonSupportFixture/FilterAndSortStoredCandidatesWithCommonPrefix/4096 2724379 ns 2718199 ns 264 PythonSupportFixture/FilterAndSortStoredCandidatesWithCommonPrefix/8192 6263308 ns 6267897 ns 112 PythonSupportFixture/FilterAndSortStoredCandidatesWithCommonPrefix/16384 12994143 ns 12792082 ns 50 PythonSupportFixture/FilterAndSortStoredCandidatesWithCommonPrefix/32768 27466364 ns 27300175 ns 28 PythonSupportFixture/FilterAndSortStoredCandidatesWithCommonPrefix/65536 57326742 ns 56727636 ns 11 PythonSupportFixture_BigO 54.99 NlgN 54.45 NlgN PythonSupportFixture_RMS 2 % 2 % ``` We see that filtering and sorting is much slower when candidates are not already stored so this is something we need to consider when trying to improve performance.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/ycmd/818)
ycm-core · Aug 25, 2017 · 4480708 · 4480708
2 parents 2575bdb + 6ed6803
commit 4480708
Show file tree

Hide file tree

Showing 5 changed files with 180 additions and 12 deletions.
diff --git a/cpp/ycm/PythonSupport.h b/cpp/ycm/PythonSupport.h
@@ -18,6 +18,8 @@
 #ifndef PYTHONSUPPORT_H_KWGFEX0V
 #define PYTHONSUPPORT_H_KWGFEX0V
 
+#include "DLLDefines.h"
+
 #include <boost/python.hpp>
 
 namespace YouCompleteMe {
@@ -26,7 +28,7 @@ namespace YouCompleteMe {
 /// python list |candidates|, a |candidate_property| on which to filter and sort
 /// the candidates and a user query, returns a new sorted python list with the
 /// original objects that survived the filtering.
-boost::python::list FilterAndSortCandidates(
+YCM_DLL_EXPORT boost::python::list FilterAndSortCandidates(
   const boost::python::list &candidates,
   const std::string &candidate_property,
   const std::string &query );

diff --git a/cpp/ycm/benchmarks/BenchUtils.cpp b/cpp/ycm/benchmarks/BenchUtils.cpp
@@ -0,0 +1,40 @@
+// Copyright (C) 2017 ycmd contributors
+//
+// This file is part of ycmd.
+//
+// ycmd is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// ycmd is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with ycmd.  If not, see <http://www.gnu.org/licenses/>.
+
+#include "BenchUtils.h"
+
+namespace YouCompleteMe {
+
+std::vector< std::string > GenerateCandidatesWithCommonPrefix(
+  const std::string prefix, int number ) {
+
+  std::vector< std::string > candidates;
+
+  for ( int i = 0; i < number; ++i ) {
+    std::string candidate = "";
+    int letter = i;
+    for ( int pos = 0; pos < 5; letter /= 26, ++pos ) {
+      candidate = std::string( 1, letter % 26 + 'a' ) + candidate;
+    }
+    candidate = prefix + candidate;
+    candidates.push_back( candidate );
+  }
+
+  return candidates;
+}
+
+} // namespace YouCompleteMe
diff --git a/cpp/ycm/benchmarks/BenchUtils.h b/cpp/ycm/benchmarks/BenchUtils.h
@@ -0,0 +1,32 @@
+// Copyright (C) 2017 ycmd contributores
+//
+// This file is part of ycmd.
+//
+// ycmd is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// ycmd is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with ycmd.  If not, see <http://www.gnu.org/licenses/>.
+
+#ifndef BENCHUTILS_H_7UY2GEP1
+#define BENCHUTILS_H_7UY2GEP1
+
+#include <string>
+#include <vector>
+
+namespace YouCompleteMe {
+
+// Generate a list of |number| candidates of the form |prefix|[a-z]{5}.
+std::vector< std::string > GenerateCandidatesWithCommonPrefix(
+  const std::string prefix, int number );
+
+} // namespace YouCompleteMe
+
+#endif /* end of include guard: BENCHUTILS_H_7UY2GEP1 */
diff --git a/cpp/ycm/benchmarks/IdentifierCompleter_bench.cpp b/cpp/ycm/benchmarks/IdentifierCompleter_bench.cpp
@@ -16,6 +16,7 @@
 // along with ycmd.  If not, see <http://www.gnu.org/licenses/>.
 
 #include "benchmark/benchmark_api.h"
+#include "BenchUtils.h"
 #include "CandidateRepository.h"
 #include "IdentifierCompleter.h"
 
@@ -31,18 +32,10 @@ class IdentifierCompleterFixture : public benchmark::Fixture {
 
 BENCHMARK_DEFINE_F( IdentifierCompleterFixture, CandidatesWithCommonPrefix )(
     benchmark::State& state ) {
-  // Generate a list of candidates of the form a_A_a_[a-z]{5}.
-  std::vector< std::string > candidates;
-  for ( int i = 0; i < state.range( 0 ); ++i ) {
-    std::string candidate = "";
-    int letter = i;
-    for ( int pos = 0; pos < 5; letter /= 26, ++pos ) {
-      candidate = std::string( 1, letter % 26 + 'a' ) + candidate;
-    }
-    candidate = "a_A_a_" + candidate;
-    candidates.push_back( candidate );
-  }
 
+  std::vector< std::string > candidates;
+  candidates = GenerateCandidatesWithCommonPrefix( "a_A_a_",
+                                                   state.range( 0 ) );
   IdentifierCompleter completer( candidates );
 
   while ( state.KeepRunning() )
@@ -51,6 +44,7 @@ BENCHMARK_DEFINE_F( IdentifierCompleterFixture, CandidatesWithCommonPrefix )(
   state.SetComplexityN( state.range( 0 ) );
 }
 
+
 BENCHMARK_REGISTER_F( IdentifierCompleterFixture, CandidatesWithCommonPrefix )
     ->RangeMultiplier( 2 )
     ->Range( 1, 1 << 16 )

diff --git a/cpp/ycm/benchmarks/PythonSupport_bench.cpp b/cpp/ycm/benchmarks/PythonSupport_bench.cpp
@@ -0,0 +1,100 @@
+// Copyright (C) 2017 ycmd contributors
+//
+// This file is part of ycmd.
+//
+// ycmd is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// ycmd is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with ycmd.  If not, see <http://www.gnu.org/licenses/>.
+
+#include "benchmark/benchmark_api.h"
+#include "BenchUtils.h"
+#include "CandidateRepository.h"
+// iostream is included because of a bug with Python earlier than 2.7.12
+// and 3.5.3 on OSX and FreeBSD.
+#include <iostream>
+#include "PythonSupport.h"
+
+namespace YouCompleteMe {
+
+class PythonSupportFixture : public benchmark::Fixture {
+public:
+  void SetUp( const benchmark::State& state ) {
+    CandidateRepository::Instance().ClearCandidates();
+  }
+};
+
+
+BENCHMARK_DEFINE_F( PythonSupportFixture,
+                    FilterAndSortUnstoredCandidatesWithCommonPrefix )(
+    benchmark::State& state ) {
+
+  std::vector< std::string > raw_candidates;
+  raw_candidates = GenerateCandidatesWithCommonPrefix( "a_A_a_",
+                                                       state.range( 0 ) );
+
+  boost::python::list candidates;
+  for ( auto insertion_text : raw_candidates ) {
+    boost::python::dict candidate;
+    candidate[ "insertion_text" ] = insertion_text;
+    candidates.append( candidate );
+  }
+
+  while ( state.KeepRunning() ) {
+    state.PauseTiming();
+    CandidateRepository::Instance().ClearCandidates();
+    state.ResumeTiming();
+    FilterAndSortCandidates( candidates, "insertion_text", "aA" );
+  }
+
+  state.SetComplexityN( state.range( 0 ) );
+}
+
+
+BENCHMARK_DEFINE_F( PythonSupportFixture,
+                    FilterAndSortStoredCandidatesWithCommonPrefix )(
+    benchmark::State& state ) {
+
+  std::vector< std::string > raw_candidates;
+  raw_candidates = GenerateCandidatesWithCommonPrefix( "a_A_a_",
+                                                       state.range( 0 ) );
+
+  boost::python::list candidates;
+  for ( auto insertion_text : raw_candidates ) {
+    boost::python::dict candidate;
+    candidate[ "insertion_text" ] = insertion_text;
+    candidates.append( candidate );
+  }
+
+  // Store the candidates in the repository.
+  FilterAndSortCandidates( candidates, "insertion_text", "aA" );
+
+  while ( state.KeepRunning() )
+    FilterAndSortCandidates( candidates, "insertion_text", "aA" );
+
+  state.SetComplexityN( state.range( 0 ) );
+}
+
+
+BENCHMARK_REGISTER_F( PythonSupportFixture,
+                      FilterAndSortUnstoredCandidatesWithCommonPrefix )
+    ->RangeMultiplier( 2 )
+    ->Range( 1, 1 << 16 )
+    ->Complexity();
+
+
+BENCHMARK_REGISTER_F( PythonSupportFixture,
+                      FilterAndSortStoredCandidatesWithCommonPrefix )
+    ->RangeMultiplier( 2 )
+    ->Range( 1, 1 << 16 )
+    ->Complexity();
+
+} // namespace YouCompleteMe