Packaging of journalbeat #8702

kvch · 2018-10-23T15:31:26Z

Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187).

Minor changes:

add missing X-Pack folder to journalbeat
do not crosscompile journalbeat due to missing dependencies locally

* add x-pack files * install required libsystemd-dev * build platforms linux/amd64 using Debian 8

urso · 2018-10-23T17:38:02Z

@kvch The journaldbeat build failed on travis. You missing systemd dev packages (header file is missing)?

Do we need to call apt-get update whenever we want to cross compile?

andrewkroh · 2018-10-23T17:46:49Z

dev-tools/mage/crossbuild.go

@@ -167,6 +167,10 @@ func crossBuildImage(platform string) (string, error) {
 		// Use an older version of libc to gain greater OS compatibility.
 		// Debian 7 uses glibc 2.13.
 		tagSuffix = "main-debian7"
+		// journalbeat requires Debian 8 due to systemd support.


Nearly all logic in the mage package is agnostic of the beat names. I think you keep this logic in the journalbeat magefile if you add a new CrossBuildOption that allows you to pass in your own "ImageSelector" func. If none is set then it should default to using mage.crossBuildImage.

type ImageSelectorFunc func(platform string) (string, error) func ImageSelector(f ImageSelectorFunc) func(params *crossBuildParams) { return func(params *crossBuildParams) { params.ImageSelector = f } }

andrewkroh · 2018-10-23T17:55:39Z

journalbeat/magefile.go

+		return err
+	}
+
+	return sh.Run("apt-get", "install", "-y", "--no-install-recommends", "libsystemd-dev")


Does this work for linux/386? I'd assume that you need to run specific commands for each architecture like apt-get install libsystemd-dev:i386.

Then for armv7 and all other architectures you'd need something like dpkg --add-architecture armhf && apt-get update && apt-get install libsystemd-dev:armhf.

andrewkroh · 2018-10-23T18:01:06Z

libbeat/scripts/Makefile

@@ -106,9 +106,11 @@ ${BEAT_NAME}.test: $(GOFILES_ALL)
 .PHONY: crosscompile
 crosscompile: ## @build Cross-compile beat for the OS'es specified in GOX_OS variable. The binaries are placed in the build/bin directory.
 crosscompile: $(GOFILES)
+ifneq ($(shell [[ $(BEAT_NAME) == journalbeat ]] && echo true ),true)


Do we need this? How about removing the thing that's invoking crosscompile for journalbeat (I assume this can just be dropped from the travis config).

crosscompile was not added to Travis, so there was nothing to remove. I added the conditional here, so if someone tries to run it locally, it is skipped.

dev-tools/mage/settings.go

dev-tools/mage/godaemon.go

kvch · 2018-10-24T11:47:35Z

I updated the PR with the platform specific packages. Unfortunately, there are no deb package for ppc64 and mips64 architectures at all.

kvch · 2018-10-24T12:36:58Z

Failing tests in Jenkins are unrelated.

ph · 2018-10-24T12:42:58Z

I've tested it locally and I can see both OSS and Xpack packages and everything look fine in them.

They are two other things that you need to change:

Makefile at https://github.com/elastic/beats/blob/master/Makefile#L3
Root magefile.go https://github.com/elastic/beats/blob/master/magefile.go#L36

ph

Makefile and magefile at the root of beats need some editing.

kvch · 2018-10-24T12:45:35Z

@ph Updated the PR.

tsg

Ran a local mage and everything seems to work fine from what I could see at a cursory look.

andrewkroh · 2018-10-24T13:09:22Z

dev-tools/mage/crossbuild.go

@@ -111,10 +122,10 @@ func CrossBuild(options ...CrossBuildOption) error {
 		if !buildPlatform.Flags.CanCrossBuild() {
 			return fmt.Errorf("unsupported cross build platform %v", buildPlatform.Name)
 		}
-		builder := GolangCrossBuilder{buildPlatform.Name, params.Target, params.InDir}
+		builder := GolangCrossBuilder{buildPlatform.Name, params.Target, params.InDir, params.ImageSelector}


andrewkroh · 2018-10-24T13:14:57Z

dev-tools/mage/godaemon.go

+
+// CrossBuildGoDaemonWithSelectableImage cross-builds the go-daemon binary
+// using the selected image.
+func CrossBuildGoDaemonWithSelectableImage(f ImageSelectorFunc) error {


Instead of adding a new function I think you could change CrossBuildGoDaemon to be variadic and pass options that are appended to the defaults. Like func CrossBuildGoDaemon(opts ...CrossBuildOption) error.

andrewkroh · 2018-10-24T13:23:12Z

journalbeat/magefile.go

+		"linux/386":      installLinux386,
+		"linux/amd64":    installDefaultLinux,
+		"linux/arm64":    installLinuxArm64,
+		"linux/armv5":    installLinuxArmLe,


I think the "le" is supposed to be "el" here. https://wiki.debian.org/ArmEabiPort

I used the golang "convention" of abbreviating little endian as le.

andrewkroh · 2018-10-24T13:24:36Z

journalbeat/magefile.go

+	deps = map[string]func() error{
+		"linux/386":      installLinux386,
+		"linux/amd64":    installDefaultLinux,
+		"linux/arm64":    installLinuxArm64,


Namewise I think that all the architecture acronyms should be in all caps, like installLinuxARM64.

andrewkroh · 2018-10-24T13:29:19Z

journalbeat/magefile.go


 	"github.com/elastic/beats/dev-tools/mage"
 )

 func init() {
 	mage.BeatDescription = "Journalbeat ships systemd journal entries to Elasticsearch or Logstash."

-	// TODO filter platforms
+	mage.Platforms = mage.Platforms.Filter("linux")


Given then ppc64 and mips64 can't be build at , I'm wondering if we should filter them like linux !linux/ppc64 !linux/mips64. The downside is that it will silently ignore those platforms, but the upside is that you can build the all of beats with PLATFORMS='+all linux/mips' make snapshot and it will just work.

kvch · 2018-10-24T14:58:11Z

jenkins test this

andrewkroh · 2018-10-24T14:55:33Z

journalbeat/magefile.go

+		switch args.OS {
+		case "linux":
+			if args.Types[0] == mage.Deb {
+				args.Spec.AddDebDependency(currentLibsystemdDev)


I wouldn't expect most systems to have -dev packages installed. I think you want a dependency on systemd.

sorry, i did not want to push this code yet

andrewkroh · 2018-10-24T14:59:34Z

journalbeat/magefile.go

+)
+
+const (
+	currentLibsystemdDev = "libsystemd-dev"


"current" and "prev" don't convey much information to me. Do names like libsystemdDevPkgNameCurrent and libsystemdDevPkgNameDebian8 match up with their expected usage?

I removed the second const because on Debian 8 libsystemd-dev is already available.

andrewkroh · 2018-10-24T15:18:06Z

journalbeat/magefile.go

+		"linux/386":      installLinux386,
+		"linux/amd64":    installLinuxAMD64,
+		"linux/arm64":    installLinuxARM64,
+		"linux/armv5":    installLinuxARMLe,


s/installLinuxs390x/installLinuxS390X/

Actually, can you just make everything after "Linux" all-caps for each function. Most everything there is an acronym so with Go's naming conventions I think they should all be in caps. Plus there's a mix of "Le" and "le" which is inconsistent.

andrewkroh

LGTM.

I tested it with

PLATFORMS='+all linux/386 linux/amd64 linux/arm64 linux/armv6 linux/armv7 linux/mips linux/mipsle linux/mips64le linux/ppc64le linux/s390x' mage -v package

I installed the x86_64 RPM. Nice to see it's using dlopen.

# ldd /usr/share/journalbeat/bin/journalbeat
	linux-vdso.so.1 =>  (0x00007fff4b7f0000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f9c8db6f000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9c8d952000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f9c8d5be000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9c8dd73000)

ph

LGTM good change @kvch ! :)

kvch · 2018-10-24T17:10:55Z

Failing test are unrelated.
Opened issue about missing dependency: https://github.com/elastic/infra/issues/7529

Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally

* Initialize Journalbeat (#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (#8324) ### Matching support From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` ### Docker fields Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` ### Parse timestamp of entries Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. ### Save custom fields Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. ### Fields && processors From now on it is possible to configure `processors` and `fields`, etc on `input` level. ### Metrics The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (#8618) * Packaging of journalbeat (#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (#8735) * Add journalbeat docs

* Initialize Journalbeat (elastic#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (elastic#8324) From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. From now on it is possible to configure `processors` and `fields`, etc on `input` level. The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (elastic#8618) * Packaging of journalbeat (elastic#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (elastic#8735) * Add journalbeat docs (cherry picked from commit 24d0e08)

* Add Journalbeat (#8703) * Initialize Journalbeat (#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (#8324) From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. From now on it is possible to configure `processors` and `fields`, etc on `input` level. The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (#8618) * Packaging of journalbeat (#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (#8735) * Add journalbeat docs (cherry picked from commit 24d0e08)

* Initialize Journalbeat (elastic#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (elastic#8324) ### Matching support From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` ### Docker fields Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` ### Parse timestamp of entries Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. ### Save custom fields Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. ### Fields && processors From now on it is possible to configure `processors` and `fields`, etc on `input` level. ### Metrics The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (elastic#8618) * Packaging of journalbeat (elastic#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (elastic#8735) * Add journalbeat docs

* Add Journalbeat (elastic#8703) * Initialize Journalbeat (elastic#8277) This is the first PR to initialize Journalbeat with minimal functionality. The architecture is mimicing Filebeat so it can be merged into FB in the future. It means it has multiple inputs which can share configuration (`backoff`, `backoff_factor`, etc.). Inputs can have multiple readers, each reader reads from a journal specified in the list of `paths`. The readers are not going to implement the interface `Harverster` until it's merged into Filebeat, because it would overcomplicate event publishing unnecessarily and would need to duplicate too much Filebeat code. Checkpointing is copied from Winlogbeat. Once the new registry file is merged, it will be migrated. Example configuration to read from the beginning of the local journal ```yml journalbeat.inputs: - paths: [] seek: head ``` Features * read from local journal, journal file and directory * position tracking by using check-pointing as it's done in Winlogbeat * seek to "tail", "head", "cursor" * minimal E2E tests * fields.yml and documentation Vendored: * github.com/coreos/go-systemd/sdjournal * Journalbeat matches support && minor additions (elastic#8324) From now on it's possible to match for journal entry fields in Journalbeat using the new option `matches`. This requires a list of key value pairs separated by "=". The key has to be a journalbeat event key (e.g systemd.unit) and the value is the exact value journal reader needs to find in the entries. Example configuration which returns NGINX and dhclient entries from the journal: ```yml include_matches: - "systemd.unit=nginx" - "process.name=dhclient" ``` Added docker fields from: https://docs.docker.com/config/containers/logging/journald/ - `container.id` - `container.id_truncated` - `container.name` - `container.image.tag` - `container.partial` Journalbeat parses the timestamp of the entry and adds it to the event as `@timestamp`. The time of reading by Journalbeat is saved in `read_timestamp`. Custom fields by various sources are stored under `custom`. Field names are normalized, meaning `"_"` prefix is removed and every letter is lowercase. From now on it is possible to configure `processors` and `fields`, etc on `input` level. The size of each open reader is reporting in bytes: ``` { "journalbeat": { "journals": { "journal_1": { "path": "system.journal", "size_in_bytes": 123124214, } } } ``` * Minor improvements to Journalbeat (elastic#8618) * Packaging of journalbeat (elastic#8702) Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally * Add journalbeat docs (elastic#8735) * Add journalbeat docs (cherry picked from commit 24d0e08)

Journalbeat is going to be built using the new Debian 8 container, because systemd version in Debian 7 is too old (v44 instead of the required v187). Minor changes: * add missing X-Pack folder to journalbeat * do not crosscompile journalbeat due to missing dependencies locally

kvch added 3 commits October 19, 2018 09:32

add journalbeat x-pack files

d355ef8

build Journalbeat using mage

bccfee3

* add x-pack files * install required libsystemd-dev * build platforms linux/amd64 using Debian 8

do not crosscompile journalbeat

9197714

kvch added the review label Oct 23, 2018

kvch requested review from urso and andrewkroh October 23, 2018 15:31

kvch mentioned this pull request Oct 23, 2018

Add Journalbeat #8703

Merged

andrewkroh requested changes Oct 23, 2018

View reviewed changes

ph self-requested a review October 23, 2018 23:45

add missing dependency

f606389

urso removed their request for review October 24, 2018 08:35

kvch added 2 commits October 24, 2018 13:31

selectable docker image

6068254

install packages by platform

d717e15

houndci-bot reviewed Oct 24, 2018

View reviewed changes

dev-tools/mage/settings.go Show resolved Hide resolved

dev-tools/mage/godaemon.go Outdated Show resolved Hide resolved

feed the hound

9924045

ph suggested changes Oct 24, 2018

View reviewed changes

add journalbeat to makefiles under project root

27ae222

tsg approved these changes Oct 24, 2018

View reviewed changes

kvch added 2 commits October 24, 2018 15:13

filter out platforms without appropiate packages

2a4ef65

fix deps

8d2181f

andrewkroh reviewed Oct 24, 2018

View reviewed changes

kvch added 2 commits October 24, 2018 16:40

final fix of deps

49126c0

address review notes

a6892e3

andrewkroh reviewed Oct 24, 2018

View reviewed changes

kvch added 3 commits October 24, 2018 17:03

rm unfinished code

4c936c9

rename consts && rm unnecessary case

88fcfc9

more renaming

fcb2fd9

andrewkroh reviewed Oct 24, 2018

View reviewed changes

kvch added 2 commits October 24, 2018 17:20

more renames

6072520

move customizations to the end of the file

329428d

andrewkroh approved these changes Oct 24, 2018

View reviewed changes

ph approved these changes Oct 24, 2018

View reviewed changes

kvch merged commit b58563b into elastic:feature-journalbeat Oct 24, 2018

kvch mentioned this pull request Oct 24, 2018

Cherry-pick #8703 to 6.x: Add Journalbeat #8739

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Packaging of journalbeat #8702

Packaging of journalbeat #8702

kvch commented Oct 23, 2018

urso commented Oct 23, 2018

andrewkroh Oct 23, 2018

andrewkroh Oct 23, 2018

andrewkroh Oct 23, 2018

kvch Oct 24, 2018

kvch commented Oct 24, 2018 •

edited

Loading

kvch commented Oct 24, 2018

ph commented Oct 24, 2018

ph left a comment

kvch commented Oct 24, 2018

tsg left a comment

andrewkroh Oct 24, 2018

andrewkroh Oct 24, 2018

andrewkroh Oct 24, 2018

kvch Oct 24, 2018

andrewkroh Oct 24, 2018

andrewkroh Oct 24, 2018

kvch commented Oct 24, 2018

andrewkroh Oct 24, 2018

kvch Oct 24, 2018

andrewkroh Oct 24, 2018

kvch Oct 24, 2018

andrewkroh Oct 24, 2018

andrewkroh left a comment

ph left a comment

kvch commented Oct 24, 2018

Packaging of journalbeat #8702

Packaging of journalbeat #8702

Conversation

kvch commented Oct 23, 2018

urso commented Oct 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Oct 24, 2018 • edited Loading

kvch commented Oct 24, 2018

ph commented Oct 24, 2018

ph left a comment

Choose a reason for hiding this comment

kvch commented Oct 24, 2018

tsg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvch commented Oct 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

ph left a comment

Choose a reason for hiding this comment

kvch commented Oct 24, 2018

kvch commented Oct 24, 2018 •

edited

Loading