Estafette

The estafette-foundation library is part of the Estafette CI system documented at https://estafette.io.

Please file any issues related to Estafette CI at https://github.com/estafette/estafette-ci-central/issues

Estafette-foundation

This library provides building blocks for creating

This library has contracts for requests / responses between various components of the Estafette CI system.

Development

To start development run

git clone git@github.com:estafette/estafette-foundation.git
cd estafette-foundation

Before committing your changes run

go test ./...
go mod tidy

Usage

To add this module to your golang application run

go get github.com/estafette/estafette-foundation

Initialize logging

import "github.com/estafette/estafette-foundation"

foundation.InitLogging(app, version, branch, revision, buildDate)

Initialize Prometheus metrics endpoint

import "github.com/estafette/estafette-foundation"

foundation.InitMetrics()

Handle graceful shutdown

import "github.com/estafette/estafette-foundation"

gracefulShutdown, waitGroup := foundation.InitGracefulShutdownHandling()

// your core application logic, making use of the waitGroup for critical sections

foundation.HandleGracefulShutdown(gracefulShutdown, waitGroup)

Watch mounted folder for changes

import "github.com/estafette/estafette-foundation"

foundation.WatchForFileChanges("/path/to/mounted/secret/or/configmap", func(event fsnotify.Event) {
  // reinitialize parts making use of the mounted data
})

Apply jitter to a number to introduce randomness

Inspired by http://highscalability.com/blog/2012/4/17/youtube-strategy-adding-jitter-isnt-a-bug.html you want to add jitter to a lot of parts of your platform, like cache durations, polling intervals, etc.

import "github.com/estafette/estafette-foundation"

sleepTime := foundation.ApplyJitter(30)
time.Sleep(time.Duration(sleepTime) * time.Second)

Retry

In order to retry a function you can use the Retry function to which you can pass a retryable function with signature func() error:

import "github.com/estafette/estafette-foundation"

foundation.Retry(func() error { do something that can fail })

Without passing any additional options it will by default try 3 times, with exponential backoff with jitter applied to the interval for any error returned by the retryable function.

In order to override the defaults you can pass them in with the following options:

import "github.com/estafette/estafette-foundation"

foundation.Retry(func() error { do something that can fail }, Attempts(5), DelayMillisecond(10), Fixed())

The following options can be passed in:

Option	Config property	Description
Attempts	Attempts	Sets the number of attempts the retryable function will be attempted before returning the error
DelayMillisecond	DelayMillisecond	Sets the base number of milliseconds between the retries or to base the exponential backoff delay on
ExponentialJitterBackoff	DelayType
ExponentialBackoff	DelayType
Fixed	DelayType
AnyError	IsRetryableError

Custom options

You can also override any of the config properties by passing in a custom option with signature func(*RetryConfig), which could look like:

import "github.com/estafette/estafette-foundation"

isRetryableErrorCustomOption := func(c *foundation.RetryConfig) {
  c.IsRetryableError = func(err error) bool {
    switch e := err.(type) {
      case *googleapi.Error:
        return e.Code == 429 || (e.Code >= 500 && e.Code < 600)
      default:
        return false
    }
  }
}

foundation.Retry(func() error { do something that can fail }, isRetryableErrorCustomOption)

Limit concurrency with a semaphore

To run code in a loop concurrently with a maximum of simultanuous running goroutines do the following:

import "github.com/estafette/estafette-foundation"

// limit concurrency using a semaphore
maxConcurrency := 5
semaphore := foundation.NewSemaphore(maxConcurrency)

for _, i := range items {
  // try to acquire a lock, which only succeeds if there's less than maxConcurrency active goroutines
  semaphore.Acquire()

  go func(i Item) {
    // release the lock when done with the slow task
    defer semaphore.Release()

    // do some slow work
  }(i)
}

// wait until all concurrent goroutines are done
semaphore.Wait()

If you want to run Acquire within a select statement do so as follows:

import "github.com/estafette/estafette-foundation"

// limit concurrency using a semaphore
maxConcurrency := 5
semaphore := foundation.NewSemaphore(maxConcurrency)

for _, i := range items {
  select {
  case semaphore.GetAcquireChannel() <- struct{}{}:
    // try to acquire a lock, which only succeeds if there's less than maxConcurrency active goroutines
    semaphore.Acquire()

    go func(i Item) {
      // release the lock when done with the slow task
      defer semaphore.Release()

      // do some slow work
    }(i)

  case <-time.After(1 * time.Second):
    // running the goroutines took to long, exit instead
    return
  }
}

// wait until all concurrent goroutines are done
semaphore.Wait()