Fix snake CVE #4837

rkovalik-raft · 2024-06-26T16:52:30Z

Updates Kubernetes api client-java version to 21.0.0-legacy, which fixes CVE issue related to snake.yaml. I chose the 21.0.0-legacy version because we would need to make significant code changes if we used the 21.0.0 version, since that version makes a method we rely on private instead of public.

Compiles and passes unit tests for hmda-platform and check-digit services, working on testing other services.

rkovalik-raft · 2024-07-11T19:54:08Z

Note that the higher s3mock version requires Java 17, and tests/builds still fail due to cassandra issues

…to fix ambiguous reference errors

rkovalik-raft · 2024-07-22T20:34:39Z

Note: to run this with sbt, need to use this command:
env JDK_JAVA_OPTIONS="--add-opens java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" sbt

There are still some errors in hmda-platform when running locally, but it passes tests and the docker image builds fine.

Also ratespread-calculator still has cassandra related errors that cause a test to fail.

…changes

… fix helm errors and warnings

rkovalik-raft · 2024-08-13T16:43:09Z

build.sbt

@@ -162,6 +162,7 @@ lazy val `hmda-platform` = (project in file("hmda"))
        case "logback.xml"                           => MergeStrategy.concat
        case "META-INF/MANIFEST.MF" => MergeStrategy.discard
        case PathList("META-INF", xs@_*) => MergeStrategy.concat
+        case PathList("org", "bouncycastle", xs @_*) => MergeStrategy.first


Fixes merge strategy errors, added for all services

rkovalik-raft · 2024-08-13T16:47:02Z

common/src/main/scala/hmda/util/RealTimeConfig.scala

@@ -18,7 +18,7 @@ class RealTimeConfig(val cmName: String, val ns: String) {
    val factory = new SharedInformerFactory(client)
    val informer = factory.sharedIndexInformerFor((params: CallGeneratorParams) => {
      api.listNamespacedConfigMapCall(
-        ns, null, null, null, s"metadata.name=$cmName", null, null, params.resourceVersion, null, params.timeoutSeconds, params.watch, null)
+        ns, null, null, null, s"metadata.name=$cmName", null, null, params.resourceVersion, null, null, params.timeoutSeconds, params.watch, null)


Due to the kubernetes client-java dependency upgrade, this method now has another parameter, so I added a null parameter value here

rkovalik-raft · 2024-08-13T16:49:22Z

common/src/test/scala/hmda/utils/EmbeddedS3.scala

Deleted since it was not used and relied on S3Mock dependency, which I removed to avoid some big dependency version issues

rkovalik-raft · 2024-08-13T16:51:12Z

data-browser/src/main/scala/hmda/dataBrowser/api/DataBrowserApi.scala

@@ -17,7 +17,7 @@ import io.lettuce.core.{ ClientOptions, RedisClient }
 import monix.eval.Task
 import slick.basic.DatabaseConfig
 import slick.jdbc.JdbcProfile
-import ch.megard.akka.http.cors.scaladsl.CorsDirectives._
+import ch.megard.akka.http.cors.scaladsl.CorsDirectives.cors


All the cors related changes in these files fix the "ambiguous reference errors" from the new akka dependency version.

rkovalik-raft · 2024-08-13T16:53:39Z

kubernetes/hmda-data-browser-api/templates/deployment.yaml

@@ -160,38 +160,6 @@ spec:
                  key: aws-region
            - name: BROWSER_LOG_LEVEL
              value: {{.Values.databrowser.loglevel}}
-            - name: DATABROWSER_PG_TABLE_2017


Removed because when I tried to deploy data-browser in hmda dev with a helm command I ran into errors related to missing mlartable tables in the configmap

rkovalik-raft · 2024-08-13T16:58:23Z

modified-lar/src/test/scala/hmda/publication/lar/parser/ModifiedLarPublisherSpec.scala

Commenting out these tests that rely on S3Mock, which is removed in this PR. Leaving the test files here so we can look into re-implementing them without S3Mock somehow in the future.

rkovalik-raft · 2024-08-13T16:58:56Z

project/Dependencies.scala

  lazy val slick                    = "com.typesafe.slick" %% "slick"                               % Version.slick
  lazy val slickHikariCP            = "com.typesafe.slick" %% "slick-hikaricp"                      % Version.slick
  lazy val alpakkaSlick             = "com.lightbend.akka" %% "akka-stream-alpakka-slick"           % Version.alpakka
  lazy val postgres                 = "org.postgresql"     % "postgresql"                           % Version.postgres
  lazy val h2                       = "com.h2database"     % "h2"                                   % Version.h2 % Test
  lazy val testContainers        = "org.testcontainers"         % "testcontainers"              % Version.testContainers % "test"
-  lazy val s3Mock                = "com.adobe.testing"          % "s3mock"                      % "2.1.19" % Test


Moving the version numbers into Version.scala

rkovalik-raft · 2024-08-13T17:00:51Z

project/Version.scala

@@ -37,6 +40,6 @@ object Version {
  val scalacheckShapeless    = "1.2.5"
  val diffx                  = "0.4.0"
  val log4j                  = "2.15.0"
-  val kubernetesApi          = "15.0.1"
+  val kubernetesApi          = "21.0.0-legacy"


Upgrading this past version 17 will fix the snake.yaml cve issue. I'm using 21.0.0-legacy version here because we would need to make significant code changes if we used the 21.0.0 version, since that version makes a method we rely on private instead of public.

rkovalik-raft · 2024-08-13T17:01:10Z

project/scoverage.sbt

+addSbtPlugin("org.scoverage" % "sbt-scoverage" % "2.0.10")
+addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.10.0-RC1")


Allows us to generate dependency graphs for our services

rkovalik-raft · 2024-08-13T17:01:30Z

ratespread-calculator/src/test/scala/hmda/calculator/scheduler/APORSchedulerSpec.scala

Another test that is commented out because it relies on S3Mock

PatrickGoRaft

These changes LGTM.

Pending verification in Dev environment smoke testing

rkovalik-raft · 2024-08-15T17:27:06Z

Confirmed with @Michaeldremy that the Cypress DEV tests looked good

rkovalik-raft added 4 commits June 25, 2024 18:17

Update kubernetes api version and fix bouncycastle merge strategy

f568010

Fix bouncycastle merge strategy

66f45c3

Update log versions

0e2a3e8

revert change to log4j version to fix unit tests

5227c41

rkovalik-raft marked this pull request as draft July 8, 2024 17:29

Update slf4j logging and s3mock versions

5dfccc5

rkovalik-raft added 2 commits July 18, 2024 15:15

Set local datacenter default name to datacenter1

929bcb7

Update akka and cassandraPlugin versions and make minor code changes …

144d2af

…to fix ambiguous reference errors

rkovalik-raft added 9 commits July 22, 2024 16:42

Merge with changes from master branch

d6b0876

Fix merge conflict

e1ecd8a

add JDK_JAVA_OPTIONS add opens settings to build settings

fdaf184

Remove APOR S3Mock dependant test and revert java 17 and cassandra 5 …

0fda5d8

…changes

Merge branch 'master' into fix_snake_cve

1787e41

Remove all mentions of S3Mock and tests that use it

50dc49f

Fix cors is ambiguous errors cause by akka version upgrade

0f87a60

Add back S3Mock related tests but comment out all code

325ea6e

Update deployment yamls for ratespread-calculator and data browser to…

a3af8b9

… fix helm errors and warnings

rkovalik-raft commented Aug 13, 2024

View reviewed changes

rkovalik-raft changed the title ~~(WIP) Fix snake CVE~~ Fix snake CVE Aug 13, 2024

rkovalik-raft marked this pull request as ready for review August 13, 2024 17:02

rkovalik-raft requested review from jaredb96 and PatrickGoRaft August 13, 2024 17:02

PatrickGoRaft approved these changes Aug 14, 2024

View reviewed changes

rkovalik-raft merged commit a1b358f into cfpb:master Aug 15, 2024
2 of 3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix snake CVE #4837

Fix snake CVE #4837

rkovalik-raft commented Jun 26, 2024

rkovalik-raft commented Jul 11, 2024

rkovalik-raft commented Jul 22, 2024

rkovalik-raft Aug 13, 2024

rkovalik-raft Aug 13, 2024

rkovalik-raft Aug 13, 2024

rkovalik-raft Aug 13, 2024

rkovalik-raft Aug 13, 2024

rkovalik-raft Aug 13, 2024

rkovalik-raft Aug 13, 2024

rkovalik-raft Aug 13, 2024

rkovalik-raft Aug 13, 2024

rkovalik-raft Aug 13, 2024

PatrickGoRaft left a comment

rkovalik-raft commented Aug 15, 2024

		addSbtPlugin("org.scoverage" % "sbt-scoverage" % "2.0.10")
		addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.10.0-RC1")

Fix snake CVE #4837

Fix snake CVE #4837

Conversation

rkovalik-raft commented Jun 26, 2024

rkovalik-raft commented Jul 11, 2024

rkovalik-raft commented Jul 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PatrickGoRaft left a comment

Choose a reason for hiding this comment

rkovalik-raft commented Aug 15, 2024