Cache implementation #1073

marcelofabri · 2016-12-27T04:39:16Z

Will fix #868

TODO:

Ignore cache if SwiftLint version has changed
Ignore cache if configuration has changed
Expose cache path in CLI and configuration
Ignore cache flag
Keep benchmarks when reading from cache
Tests

marcelofabri · 2016-12-27T14:33:13Z

Source/swiftlint/Commands/LintCommand.swift

            autoreleasepool {
-                if options.benchmark {
+                if let file = linter.file.path,
+                    case let hash = linter.file.contents.hash,


I'm not sure if this is enough or we should use something more sophisticated 🤔

SwiftLintBot · 2016-12-27T17:34:14Z

	5 Messages
📖	Linting Alamofire with this PR took 2.95s vs 2.95s on master (0% slower)
📖	Linting Aerial with this PR took 0.42s vs 0.41s on master (2% slower)
📖	Linting SourceKitten with this PR took 1.68s vs 1.66s on master (1% slower)
📖	Linting Moya with this PR took 0.45s vs 0.45s on master (0% slower)
📖	Linting realm-cocoa with this PR took 3.4s vs 3.37s on master (0% slower)

Generated by 🚫 danger

codecov-io · 2016-12-27T17:54:07Z

Current coverage is 82.30% (diff: 61.64%)

Merging #1073 into master will decrease coverage by 0.48%

@@             master      #1073   diff @@
==========================================
  Files           161        164     +3   
  Lines          7909       8117   +208   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           6548       6681   +133   
- Misses         1361       1436    +75   
  Partials          0          0

Powered by Codecov. Last update 480acd7...c28e308

jpsim · 2016-12-27T22:13:46Z

Do you understand why osscheck is saying that some violations were fixed by this PR?

marcelofabri · 2016-12-27T23:33:35Z

Do you understand why osscheck is saying that some violations were fixed by this PR?

Not really 🙃

marcelofabri · 2016-12-27T23:35:27Z

I've seen the log and that's because there was a crash while getting the version number (#1045)

marcelofabri · 2016-12-27T23:50:38Z

Source/SwiftLintFramework/Models/Version.swift

+            return Version(value: value)
+        }
+
+        return Version(value: "0.15.0")


this should be reverted or done in a way that make set_version updates it as well

Done in caee985

marcelofabri · 2016-12-28T03:16:19Z

Source/SwiftLintFramework/Models/LinterCache.swift

@@ -15,7 +15,7 @@ public enum LinterCacheError: Error {
    case differentConfiguration
 }

-public struct LinterCache {
+public class LinterCache {


I had to change it to be a class to make sure all Linters used the same reference

marcelofabri · 2016-12-28T21:02:48Z

I think this is ready for review, other than missing changelog entry and README update ~~(and the version workaround)~~

# Conflicts: # Source/swiftlint/Commands/LintCommand.swift # Source/swiftlint/Extensions/Configuration+CommandLine.swift # SwiftLint.xcodeproj/project.pbxproj

# Conflicts: # Source/swiftlint/Commands/LintCommand.swift

jpsim · 2017-01-05T07:17:42Z

Source/SwiftLintFramework/Models/Version.swift

+//  Copyright © 2016 Realm. All rights reserved.
+//
+
+import Foundation


This whole file should just be public let version = "0.15.0". There's no point in getting the version from the bundle and then fall back on a string literal in case that fails if the values will always match and it'd be a bug for them not to...

The template idea that's rendered into this file via make is 👌 though.

👍 for not reading from the bundle, but I think it'd be better if we kept this namespaced inside a struct so we can avoid using String everywhere. This would also make it easier to implement version pinning if we want.

jpsim

Mostly cosmetic requests now that a global location is used for the cache

jpsim · 2017-01-10T06:21:30Z

Source/SwiftLintFramework/Models/Configuration.swift

@@ -47,6 +48,8 @@ public struct Configuration: Equatable {
    public let rules: [Rule]
    public var rootPath: String?              // the root path to search for nested configurations
    public var configurationPath: String?     // if successfully loaded from a path
+    public var hash: Int?


Why not make Configuration conform to Hashable? http://swiftdoc.org/v3.0/protocol/Hashable/

Would require naming this variable as hashValue.

Mainly because I thought it'd be too much work to make Rule conform to Hashable as well. Do you think it's worth the extra complexity?

A Rule is uniquely identified by its identifier, so its hashValue can just be identifier.hashValue and this can be implemented as a protocol extension.

What about its configuration?

Yeah, ok thinking more about it, Hashable implies Equatable, and the protocol semantics require that all user-visible data from two equal types be indistinguishable, so makes sense not to conform to it.

So, to make sure we're in the same page, do you still think we should make Configuration conform to Hashable?

jpsim · 2017-01-10T06:23:00Z

Source/SwiftLintFramework/Models/Configuration.swift

@@ -179,6 +185,7 @@ public struct Configuration: Equatable {
            }
            self.init(dict: dict)!
            configurationPath = fullPath
+            hash = yamlContents.hash


instead of hashing the yaml contents, we could hash the properties of this struct, meaning that changes to the configuration file like reordering items or adding comments wouldn't invalidate the cache.

this would also allow Configuration to conform to Hashable and for caching to work with configurations created without a YAML source.

Forget the bit about conforming to Hashable, but do you think you could adapt this to use the has value from dict at least? Would allow writing comments and reordering YAML contents without throwing away the cache.

done in c28e308, but had to bridge to NSDictionary

jpsim · 2017-01-10T06:24:45Z

Source/SwiftLintFramework/Models/Linter.swift

@@ -78,6 +84,11 @@ public struct Linter {
            deprecatedToValidIdentifier[key] = value
        }

+        if let cache = cache, let file = file.path {


let path = file.path would disambiguate from the property

jpsim · 2017-01-10T06:25:21Z

Source/SwiftLintFramework/Models/Linter.swift

@@ -86,8 +97,34 @@ public struct Linter {
        return (violations, ruleTimes)
    }

-    public init(file: File, configuration: Configuration = Configuration()!) {
+    private func cachedStyleViolations(_ benchmark: Bool = false) -> ([StyleViolation], [(id: String, time: Double)])? {


benchmark should be a required argument label to conform to Swift 3 API Design Guidelines.

jpsim · 2017-01-10T06:32:55Z

Source/SwiftLintFramework/Models/LinterCache.swift

+        lock.unlock()
+    }
+
+    public func violations(for file: String, hash: Int) -> [StyleViolation]? {


weakly typed argument. Label should be forFile file: String.

jpsim · 2017-01-10T06:34:16Z

Source/SwiftLintFramework/Models/LinterCache.swift

+    private let lock = NSLock()
+
+    public init(currentVersion: Version = .current, configurationHash: Int? = nil) {
+        cache = [String: Any]()


why not just this?

cache = [ "version": currentVersion.value, "configuration_hash: configurationHash, "files": [:] ]

jpsim · 2017-01-10T06:44:12Z

Source/SwiftLintFramework/Models/LinterCache.swift

+                      configurationHash: configurationHash)
+    }
+
+    public func cacheFile(_ file: String, violations: [StyleViolation], hash: Int) {


Doesn't conform to Swift 3 API Design Guidelines, and implementation could be simplified:

public func cache(violations: [StyleViolation], forFile file: String, fileHash: Int) { lock.lock() var filesCache = (cache["files"] as? [String: Any]) ?? [:] filesCache[file] = [ "violations": violations.map(dictionaryForViolation), "hash": fileHash ] cache["files"] = filesCache lock.unlock() }

Adds an insignificant amount of work inside the lock.

jpsim · 2017-01-10T06:45:27Z

Source/SwiftLintFramework/Models/LinterCache.swift

+        try json.write(to: url, atomically: true, encoding: .utf8)
+    }
+
+    private func dictionaryForViolation(_ violation: StyleViolation) -> [String: Any] {


Swift 3 API Design Guidelines: dictionary(for violation: StyleViolation)

jpsim · 2017-01-10T06:45:56Z

Source/SwiftLintFramework/Models/LinterCache.swift

+}
+
+extension StyleViolation {
+    fileprivate static func fromCache(_ cache: [String: Any], file: String) -> StyleViolation? {


Swift 3 API Design Guidelines: from(cache: [String: Any], file: String) -> StyleViolation?

jpsim · 2017-01-10T06:50:43Z

Source/SwiftLintFramework/Models/Configuration.swift

@@ -35,6 +35,7 @@ private enum ConfigurationKey: String {
    case swiftlintVersion = "swiftlint_version"
    case useNestedConfigs = "use_nested_configs" // deprecated
    case warningThreshold = "warning_threshold"
+    case cachePath = "cache_path"


the rest of this list is sorted

jpsim · 2017-01-10T20:09:44Z

What in the world happened here? https://circleci.com/gh/realm/SwiftLint/82

Building branch
Cloning Alamofire
Linting 5 iterations of Alamofire with branch: 1..2..3..4..5
Cloning Aerial
Linting 5 iterations of Aerial with branch: 1..2..3..4..5
Cloning SourceKitten
Linting 5 iterations of SourceKitten with branch: 1..2..3..4..5
Cloning Moya
Linting 5 iterations of Moya with branch: 1..2..3..4..5
Cloning realm-cocoa
Linting 5 iterations of realm-cocoa with branch: 1..2..3..4..5
Switched to branch 'master'
From github.com:realm/SwiftLint
   1d2b3fd..480acd7  master     -> origin/master
Building master
<unknown>:0: error: build had 1 command failures
error: exit(1): /Applications/Xcode-8.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-build-tool -f /Users/distiller/SwiftLint/.build/debug.yaml
Linting 5 iterations of Alamofire with master: 1..2..3..4..5
Linting 5 iterations of Aerial with master: 1..2..3..4..5
Linting 5 iterations of SourceKitten with master: 1..2..3..4..5
Linting 5 iterations of Moya with master: 1..2..3..4..5
Linting 5 iterations of realm-cocoa with master: 1..2..3..4..5

marcelofabri · 2017-01-10T20:15:49Z

What in the world happened here?

I have no idea 😱

jpsim · 2017-01-10T20:20:49Z

Stepping back a bit. Wouldn't we be better off just building the cache at the very end of the lint process, from a single thread, avoiding the need for locking? Having this at the Linter level is nice architecturally, but maybe we should split that up into a new Linter model that coordinates many SingleFileLinters (aka what is Linter today).

jpsim · 2017-01-10T20:24:05Z

LOL. I understand why OSSCheck is reporting massive perf regressions. It somehow can't build master, so it's running this PR thinking that it's master after things have been cached from the first few branch runs. So #1147 is biting us here.

But also, we should disable the cache when running OSSCheck.

marcelofabri · 2017-01-10T20:25:16Z

Stepping back a bit. Wouldn't we be better off just building the cache at the very end of the lint process, from a single thread, avoiding the need for locking?

Maybe, this implementation started before we had multithread lint. We could measure the lock cost to see if it's worth the refactoring and the "architectural cost".

jpsim · 2017-01-10T22:20:22Z

Maybe, this implementation started before we had multithread lint. We could measure the lock cost to see if it's worth the refactoring and the "architectural cost".

Let's get this in, and look into avoiding locking later.

Awesome work! 🎉

jpsim · 2017-01-10T22:28:41Z

Source/SwiftLintFramework/Models/Configuration.swift

@@ -179,6 +185,7 @@ public struct Configuration: Equatable {
            }
            self.init(dict: dict)!
            configurationPath = fullPath
+            hash = dict.bridge().hashValue


This is returning a value of 3 for Realm. Seems suspicious: https://github.com/realm/realm-cocoa/blob/master/.swiftlint.yml

maybe we should revert to use the String hash then?

Nah, apparently NSDictionary hash just returns the count. I've tested dict.description.hashValue locally and that seems to work surprisingly well.

Maintains the same value if items are re-ordered, and if comments are added/edited.

jpsim · 2017-01-10T22:46:46Z

Source/SwiftLintFramework/Models/Configuration.swift

@@ -269,7 +276,8 @@ private func validKeys(ruleList: RuleList) -> [String] {
        .swiftlintVersion,
        .useNestedConfigs,
        .warningThreshold,
-        .whitelistRules
+        .whitelistRules,
+        .cachePath


This list is also otherwise alphabetized.

jpsim · 2017-01-10T22:47:13Z

Source/SwiftLintFramework/Models/Linter.swift

+        return (cachedViolations, ruleTimes)
+    }
+
+    public init(file: File,


this declaration doesn't need to be broken up in multiple lines.

jpsim · 2017-01-10T22:49:14Z

Source/swiftlint/Extensions/LinterCache+CommandLine.swift

+    return path.map(URL.init(fileURLWithPath:)) ?? defaultCacheURL(options: options)
+}
+
+private func defaultCacheURL(options: LintOptions) -> URL {


Much of this should be moved into SwiftLintFramework IMHO, so that the cache is used even if not going through the SwiftLint CLI.

It'd be used, you'd just need to setup by yourself since we depend on LintOptions to resolve the cache path

Only if overriding the default. Wouldn't be much different than the path we pass to Configuration: https://github.com/realm/SwiftLint/blob/master/Source/SwiftLintFramework/Models/Configuration.swift#L162

can we do that on a later moment? 😬

* master: fix typos in ProjectMock configuration files update SourceKitten to 0.17 build SwiftLint in release mode when running benchmarks run Circle CI's Xcode 8.2 image don't pass CI if Danger crashes Fix triggering example and add couple more Use range parameter if exists CHANGELOG entries must break at the 80th char Fix indentation Update CHANGELOG Delete extra variable Clean up code and improve overall logic Add range to avoid whole file lookup Clean up regex pattern Delete print command Fix test case clause Ignore tuples Append test to allTests array Add CHANGELOG entry Add UnusedOptionalBinding rule

and group perf improvements at the top of the "Enhancements"

fail the build if it couldn't be built even after retrying

jpsim · 2017-01-11T00:37:20Z

🎉 this is gonna be 'uuuge 🎉

marcelofabri · 2017-01-11T00:39:56Z

Awesomeee 💯

marcelofabri commented Dec 27, 2016

View reviewed changes

marcelofabri commented Dec 28, 2016

View reviewed changes

marcelofabri added 20 commits December 31, 2016 00:24

Initial cache implementation

c698d02

Ignore cache when versions are different

a92a24c

Fixing compile isse

f1e3376

Add some tests

b1f62e2

Add more tests

9d42a73

Ignore cache if configuration has changed

a47385a

Improved cache structure

77e4249

Temporary fix for getting version on SPM

34af12a

Moving cache usage to the linter

3d37624

Supporting CLI and configuration options

c7461e0

Merge remote-tracking branch 'upstream/master' into cache

b006a28

# Conflicts: # Source/swiftlint/Commands/LintCommand.swift # Source/swiftlint/Extensions/Configuration+CommandLine.swift # SwiftLint.xcodeproj/project.pbxproj

Merge remote-tracking branch 'upstream/master' into cache

1ef461b

Merge remote-tracking branch 'upstream/master' into cache

a2ea9ca

# Conflicts: # Source/swiftlint/Commands/LintCommand.swift

Make LinterCache final

b3cd500

Move cache code to an extension

a56dba7

Merge remote-tracking branch 'upstream/master' into cache

b9eb54a

Merge remote-tracking branch 'upstream/master' into cache

119c214

Make LinterCache thread-safe

e8ead88

Merge branch 'master' into cache

8af6552

Generate Version.swift from make set_version

caee985

jpsim reviewed Jan 5, 2017

View reviewed changes

Save cache in shared folder by default

5e30935

jpsim requested changes Jan 10, 2017

View reviewed changes

jpsim reviewed Jan 10, 2017

View reviewed changes

marcelofabri added 2 commits January 10, 2017 12:54

Merge branch 'master' into cache

e1e2369

PR feedback

94f48f4

marcelofabri mentioned this pull request Jan 10, 2017

Segmentation fault on trailing_comma with Swift 2.3 #921

Closed

Merge branch 'master' into cache

b7b905c

Get hash from parsed dictionary instead of String

c28e308

jpsim approved these changes Jan 10, 2017

View reviewed changes

jpsim requested changes Jan 10, 2017

View reviewed changes

marcelofabri and others added 8 commits January 10, 2017 20:49

Update how hash is calculated

66d0bb2

PR feedback

2a97a1c

disable caching when running OSSCheck

4a9806f

remove trailing comma from array

86214a7

run LinterCacheTests on Linux

41251d8

add changelog entry for caching

b189da0

and group perf improvements at the top of the "Enhancements"

disable caching, handle errors & retry if can't build for OSSCheck

5702ef3

fail the build if it couldn't be built even after retrying

jpsim merged commit 4f225a4 into realm:master Jan 11, 2017

jpsim mentioned this pull request Jan 11, 2017

Setting goals for a 1.0.0 milestone #427

Closed

marcelofabri deleted the cache branch January 11, 2017 00:40

Cache implementation #1073

Cache implementation #1073

Conversation

marcelofabri commented Dec 27, 2016 • edited Loading

Choose a reason for hiding this comment

SwiftLintBot commented Dec 27, 2016 • edited Loading

codecov-io commented Dec 27, 2016 • edited Loading

Current coverage is 82.30% (diff: 61.64%)

jpsim commented Dec 27, 2016

marcelofabri commented Dec 27, 2016

marcelofabri commented Dec 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcelofabri commented Dec 28, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpsim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpsim commented Jan 10, 2017

marcelofabri commented Jan 10, 2017

jpsim commented Jan 10, 2017

jpsim commented Jan 10, 2017

marcelofabri commented Jan 10, 2017

jpsim commented Jan 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpsim Jan 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpsim commented Jan 11, 2017

marcelofabri commented Jan 11, 2017

marcelofabri commented Dec 27, 2016 •

edited

Loading

SwiftLintBot commented Dec 27, 2016 •

edited

Loading

codecov-io commented Dec 27, 2016 •

edited

Loading

marcelofabri commented Dec 28, 2016 •

edited

Loading

jpsim Jan 10, 2017 •

edited

Loading