Skip to content

Commit

Permalink
GitLab support using personal access tokens (#73)
Browse files Browse the repository at this point in the history
* GitLab support using personal access tokens #72

* GitLab support using personal access tokens #72

* #72 fixing PR comments

* #72 fixing PR comments

Co-authored-by: Satish Kolli <skkolli@users.noreply.github.com>
  • Loading branch information
satish-centrifuge and skkolli authored Feb 19, 2020
1 parent c5c46fe commit cefc258
Show file tree
Hide file tree
Showing 4 changed files with 113 additions and 0 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
## [0.9 - SNAPSHOT]
- TODO: Upgrade to Kafka 2.4.x (PR welcome)
- Added Bitbucket Cloud as an ACL source
- Added GitLab as an ACL souce

## [0.8 - 10/01/2020]
- Added possibility to run AclSyncornizer just once and then stop KSM (Issue #56)
Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ User:peter,Cluster,LITERAL,kafka-cluster,Create,Allow,*
Current sources shipping with KSM include:
- File
- GitHub
- GitLab (using Personal Auth Tokens)
- BitBucket
- Amazon S3
- Build your own (and contribute back!)
Expand Down Expand Up @@ -128,6 +129,13 @@ The [default configurations](src/main/resources/application.conf) can be overwri
- `com.github.simplesteph.ksm.source.NoSourceAcl` (default): No source for the ACLs. Only use with `KSM_READONLY=true`
- `com.github.simplesteph.ksm.source.FileSourceAcl`: get the ACL source from a file on disk. Good for POC
- `com.github.simplesteph.ksm.source.GitHubSourceAcl`: get the ACL from GitHub. Great to get started quickly and store the ACL securely under version control.
- `com.github.simplesteph.ksm.source.GitLabSourceAcl`: get the ACL from GitLab using pesonal access tokens. Great to get started quickly and store the ACL securely under version control.
- `SOURCE_GITLAB_REPOID` GitLab project id
- `SOURCE_GITLAB_FILEPATH` Path to the ACL file in GitLab project
- `SOURCE_GITLAB_BRANCH` Git Branch name
- `SOURCE_GITLAB_HOSTNAME` GitLab Hostname
- `SOURCE_GITLAB_ACCESSTOKEN` GitLab Personal Access Token. See [Personal access tokens
](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) to authenticate with the GitLab API.
- `com.github.simplesteph.ksm.source.S3SourceAcl`: get the ACL from S3. Good for when you have a S3 bucket managed by Terraform or Cloudformation. This requires `region`, `bucketname` and `objectkey`. See [Access credentials](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) for credentials management.
- `SOURCE_S3_REGION` AWS S3 Region
- `SOURCE_S3_BUCKETNAME` AWS S3 Bucket name
Expand Down
11 changes: 11 additions & 0 deletions src/main/resources/application.conf
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,17 @@ source {
token = ${?SOURCE_GITHUB_TOKEN}
}
}
gitlab {
repoid = ${?SOURCE_GITLAB_REPOID}
filepath = "example/acls.csv"
filepath = ${?SOURCE_GITLAB_FILEPATH}
branch = "master"
branch = ${?SOURCE_GITLAB_BRANCH}
hostname = "gitlab.com"
// hostname for private gitlab is: $hostname/api/v4
hostname = ${?SOURCE_GITLAB_HOSTNAME}
accesstoken = ${?SOURCE_GITLAB_ACCESSTOKEN}
}
s3 {
region = ${?SOURCE_S3_REGION}
bucketname = ${?SOURCE_S3_BUCKETNAME}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
package com.github.simplesteph.ksm.source

import java.io.StringReader
import java.nio.charset.Charset
import java.util.Base64

import com.fasterxml.jackson.databind.ObjectMapper
import com.github.simplesteph.ksm.parser.AclParser
import com.typesafe.config.Config
import org.slf4j.LoggerFactory
import skinny.http.{HTTP, HTTPException, Request, Response}

import scala.util.Try

class GitLabSourceAcl extends SourceAcl {

private val log = LoggerFactory.getLogger(classOf[GitLabSourceAcl])

override val CONFIG_PREFIX: String = "gitlab"
final val REPOID_CONFIG = "repoid"
final val FILEPATH_CONFIG = "filepath"
final val BRANCH_CONFIG = "branch"
final val HOSTNAME_CONFIG = "hostname"
final val ACCESSTOKEN_CONFIG = "accesstoken"

var lastModified: Option[String] = None
val objectMapper = new ObjectMapper()
var repoid: String = _
var filepath: String = _
var branch: String = _
var hostname: String = _
var accessToken: String = _

/**
* internal config definition for the module
*/
override def configure(config: Config): Unit = {
repoid = config.getString(REPOID_CONFIG)
filepath = config.getString(FILEPATH_CONFIG)
branch = config.getString(BRANCH_CONFIG)
hostname = config.getString(HOSTNAME_CONFIG)
accessToken = config.getString(ACCESSTOKEN_CONFIG)
}

override def refresh(aclParser: AclParser): Option[SourceAclResult] = {
val url =
s"https://$hostname/api/v4/projects/$repoid/repository/files/$filepath?ref=$branch"
val request: Request = new Request(url)

// auth header
request.header("PRIVATE-TOKEN", s" $accessToken")
val metadata: Response = HTTP.head(request)
val commitId = metadata.header("X-Gitlab-Commit-Id")

log.debug(s"lastModified: ${lastModified}")
log.debug(s"commitId from Head: ${commitId}")

lastModified match {
case `commitId` =>
log.info(s"No changes were detected in the ACL file ${filepath}. Skipping .... ")
None
case _ =>
val response: Response = HTTP.get(request)
response.status match {
case 200 =>
val responseJSON = objectMapper.readTree(response.textBody)
lastModified = Some(responseJSON.get("commit_id").asText())
val b64encodedContent = responseJSON.get("content").asText()
val data = new String(
Base64.getDecoder.decode(
b64encodedContent.replace("\n", "").replace("\r", "")),
Charset.forName("UTF-8"))
// use the CSV Parser
Some(aclParser.aclsFromReader(new StringReader(data)))
case _ =>
// we got an http error so we propagate it
log.warn(response.asString)
Some(
SourceAclResult(
Set(),
List(Try(
throw HTTPException(Some("Failure to fetch file"), response)))))
}
}
}

/**
* Close all the necessary underlying objects or connections belonging to this instance
*/
override def close(): Unit = {
// HTTP
}
}

0 comments on commit cefc258

Please sign in to comment.