Skip to content

Commit

Permalink
Adds compile-time validation (#2)
Browse files Browse the repository at this point in the history
Adds a compiler-plugin that runs the smithy validators on smithy-models translated from `API` definitions found in the compiled code. 

This plugin is bundled in an artifact using assembly, as the build tools (SBT, scala-cli, ...) do not pull the transitive dependencies of compiler plugins. The bundled dependencies are, in particular, classgraph, smithy, smithy4s, smithy4s-deriving.
  • Loading branch information
Baccata authored May 12, 2024
1 parent 7de75c1 commit b4bb357
Show file tree
Hide file tree
Showing 13 changed files with 431 additions and 54 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,11 @@ jobs:

- name: Make target directories
if: github.event_name != 'pull_request' && (startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/main')
run: mkdir -p modules/core/js/target modules/core/jvm/target project/target
run: mkdir -p modules/compiler-plugin/target modules/core/js/target modules/core/jvm/target modules/bundle/target project/target

- name: Compress target directories
if: github.event_name != 'pull_request' && (startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/main')
run: tar cf targets.tar modules/core/js/target modules/core/jvm/target project/target
run: tar cf targets.tar modules/compiler-plugin/target modules/core/js/target modules/core/jvm/target modules/bundle/target project/target

- name: Upload target directories
if: github.event_name != 'pull_request' && (startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/main')
Expand Down
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,15 @@ Scala 3.4.1 or newer is required.
SBT :

```
"tech.neander" %% "smithy4s-deriving" % <version>
libraryDependencies += "tech.neander" %% "smithy4s-deriving" % <version>
addCompilerPlugin("tech.neander" %% "smithy4s-deriving-compiler" % <version>)
```

The rest of the world :
scala-cli :

```
"tech.neander::smithy4s-deriving:<version>"
//> using dep "tech.neander::smithy4s-deriving:<version>"
//> using plugin "tech.neander::smithy4s-deriving-compiler:<version>"
```

You'll typically need the following imports to use the derivation :
Expand Down Expand Up @@ -348,9 +350,15 @@ val instance = new Foo {
}
```

### Compile time validation

The `smithy4s-deriving-compiler` permits the validation of API/Schema usage within the compilation cycle. At the time of writing this, the plugin works by looking up derived `API` instances and crawling through the schemas from there, which implies that standalone `Schema` instances that are not (transitively) tied to `API` instances are not validated at compile time.



### Re-creating a smithy-model from the derived constructs (JVM only)

It is unfortunately impossible for `smithy4s-deriving` to validate, at compile time, the correct usage of the application of `@hints`. However, it is possible to automatically recreate a smithy model from the derived abstractions, and to run the smithy validators. One could use this in a unit test, for instance, to verify the correctness of their services according to the rules of smithy.
It is possible to automatically recreate a smithy model from the derived abstractions, and to run the smithy validators. One could use this in a unit test, for instance, to verify the correctness of their services according to the rules of smithy.

See an example of how to do that [here](./modules/examples/jvm/src/main/scala/printSpecs.scala).

Expand Down
57 changes: 54 additions & 3 deletions build.sbt
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
import org.eclipse.jgit.api.MergeCommand.FastForwardMode.Merge
ThisBuild / tlBaseVersion := "0.0" // your current series x.y
ThisBuild / version := {
if (!sys.env.contains("CI")) "0.0.0-SNAPSHOT"
else (ThisBuild / version).value
}

ThisBuild / organization := "tech.neander"
ThisBuild / organizationName := "Neandertech"
Expand All @@ -22,7 +27,7 @@ val smithyVersion = "1.47.0"
val smithy4sVersion = "0.18.16"
val alloyVersion = "0.3.7"

lazy val root = tlCrossRootProject.aggregate(core, examples, tests)
lazy val root = tlCrossRootProject.aggregate(core, examples, plugin, pluginBundle, tests)

lazy val core = crossProject(JVMPlatform, JSPlatform)
.in(file("modules/core"))
Expand All @@ -34,6 +39,43 @@ lazy val core = crossProject(JVMPlatform, JSPlatform)
)
)

lazy val plugin = project
.in(file("modules/compiler-plugin"))
.dependsOn(core.jvm)
.enablePlugins(AssemblyPlugin)
.settings(
name := "smithy4s-deriving-compiler-plugin",
libraryDependencies ++= Seq(
"org.scala-lang" %% "scala3-compiler" % scalaVersion.value,
"io.github.classgraph" % "classgraph" % "4.8.172",
"com.disneystreaming.smithy4s" %% "smithy4s-dynamic" % smithy4sVersion,
"com.disneystreaming.alloy" % "alloy-core" % alloyVersion
),
// ASSEMBLY
assembly / logLevel := Level.Debug,
assemblyPackageScala / assembleArtifact := false,
assembly / assemblyExcludedJars := {
val cp = (assembly / fullClasspath).value
cp.filter { x => x.data.getName.startsWith("scala3-") || x.data.getName.startsWith("jline") }
},
assemblyMergeStrategy := {
case PathList("META-INF", "smithy", "manifest") => MergeStrategy.concat
case PathList("META-INF", "services", _) => MergeStrategy.concat
case PathList("plugin.properties") => MergeStrategy.last
case x =>
val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
oldStrategy(x)
}
)

lazy val pluginBundle = project
.in(file("modules/bundle"))
.enablePlugins(AssemblyPlugin)
.settings(
name := "smithy4s-deriving-compiler",
Compile / packageBin := (plugin / assembly).value
)

lazy val tests = crossProject(JVMPlatform)
.in(file("modules/tests"))
.enablePlugins(NoPublishPlugin)
Expand All @@ -53,15 +95,24 @@ lazy val examples = crossProject(JVMPlatform, JSPlatform)
.enablePlugins(NoPublishPlugin)
.settings(
libraryDependencies ++= Seq(
"com.disneystreaming.alloy" % "alloy-core" % alloyVersion,
"com.disneystreaming.smithy4s" %% "smithy4s-http4s" % smithy4sVersion,
"com.disneystreaming.smithy4s" %% "smithy4s-dynamic" % smithy4sVersion,
"org.http4s" %% "http4s-ember-client" % "0.23.26",
"org.http4s" %% "http4s-ember-server" % "0.23.26"
)
),
autoCompilerPlugins := true,
Compile / fork := true,
Compile / scalacOptions += {
val pluginClasspath =
(plugin / Compile / fullClasspathAsJars).value.map(_.data.getAbsolutePath()).mkString(":")
s"""-Xplugin:$pluginClasspath"""
}
)
.jvmSettings(
libraryDependencies ++= Seq(
"software.amazon.smithy" % "smithy-model" % smithyVersion
)
)
.jsSettings(
Test / fork := false
)
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pluginClass=smithy4s.deriving.compiler.Smithy4sDerivingCompiler
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
/*
* Copyright 2024 Neandertech
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package smithy4s.deriving.compiler

import dotty.tools.backend.jvm.GenBCode
import dotty.tools.dotc.CompilationUnit
import dotty.tools.dotc.core.Contexts.Context
import dotty.tools.dotc.plugins.PluginPhase
import dotty.tools.dotc.plugins.StandardPlugin
import dotty.tools.dotc.report
import dotty.tools.dotc.util.NoSourcePosition
import dotty.tools.dotc.util.Spans
import io.github.classgraph.ClassGraph
import io.github.classgraph.ClassRefTypeSignature
import smithy4s.Document
import smithy4s.deriving.internals.SourcePosition
import smithy4s.dynamic.DynamicSchemaIndex
import smithy4s.dynamic.NodeToDocument
import software.amazon.smithy.model.Model
import software.amazon.smithy.model.shapes.ModelSerializer
import software.amazon.smithy.model.shapes.ShapeId as SmithyShapeId
import software.amazon.smithy.model.validation.Severity
import software.amazon.smithy.model.validation.ValidationEvent

import java.net.URLClassLoader
import java.util.Optional
import scala.jdk.CollectionConverters.*
import scala.jdk.OptionConverters.*
import scala.util.control.NonFatal

class Smithy4sDerivingCompiler extends StandardPlugin {
val name: String = "smithy4s-deriving-compiler"
override val description: String = "Runs smithy linting on derived constructs"
override def init(options: List[String]): List[PluginPhase] =
List(Smithy4sDerivingCompilerPhase())
}

class Smithy4sDerivingCompilerPhase() extends PluginPhase {

override def phaseName: String = Smithy4sDerivingCompilerPhase.name
override val runsAfter = Set(GenBCode.name)
// Overriding `runOn` instead of `run` because the latter is run per compilation unit (files)
override def runOn(units: List[CompilationUnit])(using context: Context): List[CompilationUnit] = {

val result = super.runOn(units)

val compileClasspath = context.settings.classpath.value
val output = context.settings.outputDir.value.jpath
val urls = compileClasspath.split(":").map(new java.io.File(_).toURI().toURL())
val allUrls = urls.appended(output.toUri().toURL())
val classLoader = new URLClassLoader(allUrls, this.getClass().getClassLoader())

val scanResult = new ClassGraph()
.addClassLoader(classLoader)
.enableAllInfo()
.scan()

try {
val apiClassInfo = scanResult.getClassInfo("smithy4s.deriving.API")

val builder = scanResult
.getClassesImplementing("smithy4s.deriving.API")
.filter(info => !info.isAbstract())
.asMap()
.asScala
.foldLeft(DynamicSchemaIndex.builder) { case (builder, (name, info)) =>
try {
val cls = info.loadClass(true)
val clsLocation = cls.getProtectionDomain().getCodeSource().getLocation().toURI()
// checking that the class comes from the current compilation unit
if (clsLocation == output.toUri()) {
// Getting the outer class, with the assumption that it'll be the companion object
// of the class for which an API is derived
// TODO : add some more protections
val outer = info.getOuterClasses().get(0)
val givenAPIMethodInfo = outer
.getMethodInfo()
.asScala
.find { methodInfo =>
val sig = methodInfo.getTypeSignature()
methodInfo.getParameterInfo().isEmpty && // looking for parameterless methods
sig != null &&
sig.getResultType().isInstanceOf[ClassRefTypeSignature] &&
sig.getResultType().asInstanceOf[ClassRefTypeSignature].getClassInfo() == apiClassInfo
}

val companionConstructor = outer.getConstructorInfo().get(0).loadClassAndGetConstructor()
companionConstructor.setAccessible(true)
val companion = companionConstructor.newInstance()
val givenAPIMethod = givenAPIMethodInfo.get.loadClassAndGetMethod()
val api = givenAPIMethod.invoke(companion).asInstanceOf[smithy4s.deriving.API[?]]
builder.addService[api.Free]
} else {
builder
}
} catch {
case NonFatal(e) =>
report.error(s"Error when loading ${info.getName()} ${e.getMessage()}")
e.printStackTrace()
builder
}
}

val unvalidatedModel = builder.build().toSmithyModel
val node = ModelSerializer.builder().build().serialize(unvalidatedModel)
val assemblyResult = Model
.assembler(this.getClass().getClassLoader())
.discoverModels(this.getClass().getClassLoader())
.addDocumentNode(node)
.assemble()

val events = assemblyResult.getValidationEvents().asScala
events.foreach(reportEvent(unvalidatedModel))
} finally {
scanResult.close()
}
result
}

private def reportEvent(model: Model)(event: ValidationEvent)(using context: Context): Unit = {
var message = event.getMessage()

val reason = event.getSuppressionReason().orElse(null)
if (reason != null) { message += " (" + reason + ")" }
val hint = event.getHint().orElse(null);
if (hint != null) { message += " [" + hint + "]" }

val formatted = String.format(
"%s: %s | %s",
event.getShapeId().map(_.toString).orElse("-"),
message,
event.getId()
)

val SourcePositionId = SmithyShapeId.fromParts(SourcePosition.id.namespace, SourcePosition.id.name)
val sourcePositionDecoder = Document.Decoder.fromSchema(SourcePosition.schema)

val maybeSourcePos = event
.getShapeId()
.flatMap(model.getShape)
.flatMap(sourcePos => Optional.ofNullable(sourcePos.getAllTraits().get(SourcePositionId)))
.map(_.toNode())
.map(NodeToDocument(_))
.flatMap(sourcePositionDecoder.decode(_).toOption.toJava)
.toScala

val scalaPosition = maybeSourcePos match {
case None => NoSourcePosition
case Some(pos) =>
val sourceFile = context.getSource(pos.path)
dotty.tools.dotc.util.SourcePosition(sourceFile, Spans.Span(pos.start, pos.end))
}

event.getSeverity() match
case Severity.SUPPRESSED => report.inform(formatted, scalaPosition)
case Severity.NOTE => report.inform(formatted, scalaPosition)
case Severity.WARNING => report.warning(formatted, scalaPosition)
case Severity.DANGER => report.error(formatted, scalaPosition)
case Severity.ERROR => report.error(formatted, scalaPosition)
}

}

object Smithy4sDerivingCompilerPhase {
val name = "smithy4s-deriving-compiler-phase"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
$version: "2"

namespace smithy4s.deriving.internals

@trait()
structure SourcePosition {
@required
path: String
@required
start: Integer
@required
startLine: Integer
@required
startColumn: Integer
@required
end: Integer
@required
endLine: Integer
@required
endColumn: Integer
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
internals.smithy
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
/*
* Copyright 2024 Neandertech
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package smithy4s.deriving.internals

import smithy4s.deriving.{*, given}
import smithy4s.schema.Schema
import smithy4s.ShapeTag
import smithy4s.ShapeId

case class SourcePosition(
path: String,
start: Int,
startLine: Int,
startColumn: Int,
end: Int,
endLine: Int,
endColumn: Int
) derives Schema

object SourcePosition extends ShapeTag.Companion[SourcePosition] {
val id: ShapeId = ShapeId("smithy4s.deriving.internals", "SourcePosition")
def schema: Schema[SourcePosition] = derived$Schema
}
Loading

0 comments on commit b4bb357

Please sign in to comment.