Add Stack build to Travis CI #1109

miguelpdl · 2017-07-18T14:51:21Z

Travis CI is a hosted, distributed continuous integration service used to build and test software projects. The plan is to add a continuous build of the IRATI stack to the Travis CI system

miguelpdl · 2017-07-19T11:10:04Z

Travis-CI build for the stack fails with

/home/travis/build/IRATI/stack/kernel/rnl-utils.c:3686:22: error: implicit declaration of function ‘nla_put_u64_64bit’ [-Werror=implicit-function-declaration]
                      nla_put_u64_64bit(skb_out,
                      ^

@edugrasa this line in rnl-utils was recently added for the SK_Buff work can you offer a pointer as to how to fix this up ?

edugrasa · 2017-07-19T11:40:06Z

In 2-3 days I will be doing a pull request with updated code that no longer uses netlink, so I think it's worth waiting and not solving this issue. Do you agree?

miguelpdl · 2017-07-19T12:01:29Z

Fine let me know when the pull request is being processed and I'll bring it into this branch for the Travis CI build.

dstaesse · 2017-07-31T21:02:08Z

We tried Travis a while back during PRISTINE with Marc, and one of the problems was that the free version of Travis had a build time limit, which was exceeded when trying to build a complete kernel. Now that the stack is built out-of-tree it may work. Do you have an estimate of the build time on a VM?

msune · 2017-09-08T12:16:08Z

If someone would have spare server capacity, there are ways to couple github with external CIs. Private instances of Travis CI, or gitlab CI (not sure if repo can be external) or some of the newer CIs.

miguelpdl · 2017-09-12T11:07:41Z

@dstaesse I don't have a specific build time, but I can say that I've recently managed to build it in a Docker VM on a Mac so it should be possible to build within a CI environment now.

miguelpdl · 2017-09-12T11:11:34Z

@msune I really like and have used GitLab CI. However I think the project has to be within a GitLab instance for it to be then built on an associated GLRunner. As you say though all I need is some nice offer to host a private GitLab / GLRunner instance and I can get it running up there.
Also would you have any other suggestion for a "newer" public CI ?

miguelpdl · 2017-09-12T11:19:35Z

@edugrasa a question for you in regards to the build that is hosted on Travis CI.

The build is currently failing with

/home/travis/build/IRATI/stack/kernel/rina-device.c:156:5: error: ‘IFF_PHONY_HEADROOM’ undeclared (first use in this function)
   | IFF_PHONY_HEADROOM;

From what I can tell IFF_PHONY_HEADROOM is defined in include/linux/netdevice.h, line 1381 (as a enumerator).

However this only appears in Kernel headers from v4.6 onwards.

Ubuntu 14.04 (which is one the base images for Travis CI) is on v4.4, so IFF_PHONY_HEADROOM is not defined.

Is there some way we can get around this ?

edugrasa · 2017-09-12T11:26:02Z

@miguelpdl The current implementation supports kernels versions v4.9 onwards, not sure if IFF_PHONY_HEADROOM would be the only problem in trying to support v4.6 kernels.

Is it possible to get a machine with kernel v4.9.x to do the build?

miguelpdl · 2017-09-12T11:40:52Z

Ahh okay, v4.9 onwards. But the problem is not in getting a machine and installing a newer kernel. The Travis CI environment is Ubuntu 14.04.5 LTS with kernel version: 4.4.0-93-generic which is as pointed out by the Ubuntu LTS kernel support page. Even when looking at Ubuntu 16.04 LTS support v4.6 kernel is not on the roadmap, it will be at best v4.5 in 2018.

So out of the box, at best a Ubuntu 16.10 OS would have to be in place to build the arcfire branch.

Looks like I'll have to look for an alternative CI environment.

edugrasa · 2017-09-12T11:44:08Z

I'm no expert here, but is it possible to setup Travis so that it cross-compiles for kernel v4.9? (similar to what buildroot does for building Linux images)

dstaesse · 2017-09-12T12:30:33Z

@miguelpdl The problem was never the environment, only the build time was. The C++ compilation of user space takes quite some time. If Travis/Jenkins scraps the VM after 50 minutes because it thinks it hangs, it won't build the complete stack. We need to be able to build the entire thing before the time expires.

edugrasa · 2017-09-12T12:56:38Z

Probably this is not an issue anymore? Even in a Raspberry Pi (model 3B) user-space compilation takes less than 50 minutes.. Probably the issue was total build time when the full kernel had to be compiled? This is not required anymore..

miguelpdl · 2017-09-12T14:12:32Z

Had a look at CircleCI but it too uses Ubuntu 14.04 LTS as it's base container environemnt.

@edugrasa For the moment I'd like to try and build the stack on a CI service that matches the build environment of the stack instead of going for the build-root cross-compile way.

msune · 2017-09-12T15:02:08Z

@miguelpdl I think it should be possible using Github's PUSH hooks to sync a clone of the repository in a gitlab instance, and run the CI:

https://developer.github.com/v3/repos/hooks/#test-a-push-hook

But this sounds very convoluted, and wouldn't be easily integrated with PR (reviewing etc...). There is always an option of using Travis CI payed plan (69$/month)

sandervrijders · 2017-09-12T15:24:25Z

@miguelpdl A different option may also be to download the latest kernel + headers from the Ubuntu kernel-ppa, to install them through dpkg and then install IRATI.

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.13/

msune · 2017-09-12T15:27:20Z

Can the entire IRATI code be compiled as a kernel module + libs + apps? One alternative option would be to prepare a Debian with the kernel pre-compiled, and do the module compilation only as part of the build, plus the rest of user-space and apps.

It is not ideal, as you would want to do both in-kernel and modules. But it's better than nothing.

sandervrijders · 2017-09-12T15:30:32Z

@msune The whole kernel space is compiled out of tree these days AFAIK. The problem is that the module compilation requires kernel headers that are new enough. But your point remains valid. One could for instance also only compile the userspace. Or try my suggested approach above of getting the latest kernel and headers from the kernel-ppa and trying it like that.

msune · 2017-09-12T15:32:42Z

There should be enough time to comply kernel modules + user-space only. In the old days, the entire kernel was compiled as part of the build (and the kernel was already around 42minutes or so, so timeouts were more than possible).

I have to download the code back, and give it a try with the new featurea 😄

sandervrijders · 2017-09-12T15:36:01Z

Yes, I am not worried about it timing out. I was merely stating that if it is the case that updating to kernel headers that are new enough is too difficult, one could simply do a userspace only compilation. (Since the checks for the syscalls are gone I assume) @edugrasa Can probably confirm this.

vmaffione · 2017-09-12T16:40:25Z

Hi, Some observations: 1) The compilation time now is acceptable, because we don't need to compile the whole kernel and modules, but just some kernel modules. 2) It is not necessary to build the kernel modules against the running kernel (4.4). You can build the modules against any directory with kernel sources that have been "prepared". $ wget linux-sources-4.9.tar.gz $ tar zxvf linux-sources-4.9.tar.gz $ cd linux-sources-4.9 $ make defconfig $ make modules_prepare Then you need to modify the kernel Makefile to change dir to /path/to/linux-sources-4.9 $ make -C /path/to/linux-sources-4.9 modules M= .... In the real world the ./configure is expected to provide an option to compile the kernel modules against a given local path, so that it can generate the proper makefile. See for instance what has been done for rlite https://pastebin.com/4Q67bW7c (look for --kernbuilddir and KERNBUILDDIR). Makefile template here ( https://pastebin.com/Xf6YT3CA). Cheers, Vincenzo 2017-09-12 8:46 GMT-07:00 Sander Vrijders <notifications@github.com>:

…

Yes, I am not worried about it timing out. I was merely stating that if it is the case that updating to kernel headers that are new enough is too difficult, one could simply do a userspace only compilation. (Since the checks for the syscalls are gone I assume) @edugrasa <https://github.com/edugrasa> Can probably confirm this. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1109 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEsSwfldTosWAlr5oou7tog5Ul-rBRC-ks5shqdkgaJpZM4Obd1q> .

-- Vincenzo Maffione

edugrasa · 2017-09-12T16:49:11Z

Hi Vincenzo,

I adopted that kernel configuration option from rlite to IRATI, should be the same unless I made an error :P (if so I'll fix asap). Therefore what you say should be possible in IRATI as well

vmaffione · 2017-09-12T16:52:54Z

Sure it is possible! But have you actually tried to compile the modules against external sources as outlined above? (can't try right now, sorry)

edugrasa · 2017-09-12T17:05:37Z

Not yet, maybe we can help @miguelpdl set it up

miguelpdl · 2017-09-15T10:36:45Z

@msune Yes looking at the PUSH hook that should work (Github repo to GitLab CI Runner) but it is convoluted and for sure all the PR would be missing, so not the best option. Paying for bootstrap Travis CI will not help, the environment is still a docker container with Ubuntu 14.04 LTS, so I would not have the privileges to change the kernel up to v4.9 that's needed for the build.

@miguelpdl I think it should be possible using Github's PUSH hooks to sync a clone of the repository in a gitlab instance, and run the CI:

https://developer.github.com/v3/repos/hooks/#test-a-push-hook

But this sounds very convoluted, and wouldn't be easily integrated with PR (reviewing etc...). There is always an option of using Travis CI payed plan (69$/month)

miguelpdl · 2017-09-15T10:41:06Z

@sandervrijders good option, and I see they already cover the v4.9 version needed at http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9/

However as just mentioned to Marc, since the TraviCI, and CircleCI use Docker containers with Ubuntu LTS 14.04 then I have no privileges to install the new kernel into the environment before then starting the IRATI stack build.

I will of course do this on a local machine, thanks

@miguelpdl A different option may also be to download the latest kernel + headers from the Ubuntu kernel-ppa, to install them through dpkg and then install IRATI.

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.13/

miguelpdl · 2017-09-15T10:59:49Z

@vmaffione thanks for insights, seems achievable.

I've looked at the configure and Makefile created for rlite on this case, and the use of KERNBUILDDIR is very clear.

However @edugrasa I think changes to the IRATI stack would be needed to support this.

As far as I can see in the IRATI stack build environment the KERNBUILDDIR is set in stack/kernel/Makefile.in which in turn is really set in stack/kernel/configure. But this is all set up with default values in place, with the expectation that v4.9 (or greater) kernel headers are in place. So I think some work would be needed here first to allow an option like --kernbuilddir.

Also would it be an idea to use an option --kernbuilddir in the script 'stack/install-from-scratch', more so than hidden further down in the 'stack/compile-kernel' or 'stack/kernel/configure' scripts ?

All in all I think that if this is a route to go, then a new issue (feature request) should be raised. I'll keep trying the CI build somehow through this issue.

$ wget linux-sources-4.9.tar.gz
$ tar zxvf linux-sources-4.9.tar.gz
$ cd linux-sources-4.9
$ make defconfig
$ make modules_prepare

Then you need to modify the kernel Makefile to change dir to
/path/to/linux-sources-4.9

$ make -C /path/to/linux-sources-4.9 modules M= ....

In the real world the ./configure is expected to provide an option to
compile the kernel modules against a given local path, so that it can
generate the proper makefile.
See for instance what has been done for rlite https://pastebin.com/4Q67bW7c
(look for --kernbuilddir and KERNBUILDDIR). Makefile template here (
https://pastebin.com/Xf6YT3CA).

Cheers,
Vincenzo

vmaffione · 2017-09-15T15:37:44Z

I'm surprised IRATI kernel does not compile on 4.4 or 4.6, as Linux internal APIs didn't change that much in this range What is the dependency that breaks across these versions? I personally don't think the "install-from-scratch" script is something that people (or us) should use. I would try to stick to the standard ./configure && make. 2017-09-15 3:59 GMT-07:00 Miguel Ponce de Leon <notifications@github.com>:

…

@vmaffione <https://github.com/vmaffione> thanks for insights, seems achievable. I've looked at the configure and Makefile created for rlite on this case, and the use of KERNBUILDDIR is very clear. However @edugrasa <https://github.com/edugrasa> I think changes to the IRATI stack would be needed to support this. As far as I can see in the IRATI stack build environment the KERNBUILDDIR is set in stack/kernel/Makefile.in which in turn is really set in stack/kernel/configure. But this is all set up with default values in place, with the expectation that v4.9 (or greater) kernel headers are in place. So I think some work would be needed here first to allow an option like --kernbuilddir. Also would it be an idea to use an option --kernbuilddir in the script 'stack/install-from-scratch', more so than hidden further down in the 'stack/compile-kernel' or 'stack/kernel/configure' scripts ? All in all I think that if this is a route to go, then a new issue (feature request) should be raised. I'll keep trying the CI build somehow through this issue. $ wget linux-sources-4.9.tar.gz $ tar zxvf linux-sources-4.9.tar.gz $ cd linux-sources-4.9 $ make defconfig $ make modules_prepare Then you need to modify the kernel Makefile to change dir to /path/to/linux-sources-4.9 $ make -C /path/to/linux-sources-4.9 modules M= .... In the real world the ./configure is expected to provide an option to compile the kernel modules against a given local path, so that it can generate the proper makefile. See for instance what has been done for rlite https://pastebin.com/4Q67bW7c (look for --kernbuilddir and KERNBUILDDIR). Makefile template here ( https://pastebin.com/Xf6YT3CA). Cheers, Vincenzo — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1109 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEsSwXoDEUDJlyRO_oTd6ne-5rQXrItAks5silimgaJpZM4Obd1q> .

-- Vincenzo Maffione

miguelpdl · 2017-09-18T10:28:10Z

@vmaffione okay (in regards to "install-from-scratch" script).

In regards to the IRATI kernel compile on 4.4 or 4.6. if you look at the configure script it uses $(uname -r) to locate the modules and header files, which is all fine and dandy when v4.9 the linux kernel is installed on the machine you are building with however as highlighted in the earlier comments the CI build on TravisCI is currently failing with

/home/travis/build/IRATI/stack/kernel/rina-device.c:156:5: error: ‘IFF_PHONY_HEADROOM’ undeclared (first use in this function)
   | IFF_PHONY_HEADROOM;

IFF_PHONY_HEADROOM is defined in include/linux/netdevice.h, line 1381 (as a enumerator).

However this only appears in Kernel headers from v4.6 onwards. So this will not compile with v4.4 as far as I can tell.

I'm surprised IRATI kernel does not compile on 4.4 or 4.6, as Linux
internal APIs didn't change that much in this range What is the dependency
that breaks across these versions?

I personally don't think the "install-from-scratch" script is something that people (or us) should use. I would try to stick to the standard ./configure && make.

edugrasa · 2017-09-18T12:13:42Z

Hi @miguelpdl , @vmaffione,

I'm adding the changes to be able to change the kernel build directory. It doesn't compile with v4.4 for the reason you state and also do to changes in the queueing disciplines API. I'm trying now with v4.6

  CC [M]  /root/stack/kernel/ipcps/shim-eth-utils.o
/root/stack/kernel/ipcps/shim-eth-utils.c: In function 'shim_eth_qdisc_enqueue':
/root/stack/kernel/ipcps/shim-eth-utils.c:73:9: error: too many arguments to function 'qdisc_drop'
  return qdisc_drop(skb, qdisc, to_free);
         ^
In file included from include/net/pkt_sched.h:7:0,
                 from /root/stack/kernel/ipcps/shim-eth-utils.c:24:
include/net/sch_generic.h:750:19: note: declared here
 static inline int qdisc_drop(struct sk_buff *skb, struct Qdisc *sch)
                   ^
/root/stack/kernel/ipcps/shim-eth-utils.c: In function 'shim_eth_qdisc_dequeue':
/root/stack/kernel/ipcps/shim-eth-utils.c:88:46: warning: passing argument 1 of '__qdisc_dequeue_head' from incompatible pointer type
   struct sk_buff *skb = __qdisc_dequeue_head(&qdisc->q);
                                              ^
In file included from include/net/pkt_sched.h:7:0,
                 from /root/stack/kernel/ipcps/shim-eth-utils.c:24:
include/net/sch_generic.h:607:31: note: expected 'struct Qdisc *' but argument is of type 'struct sk_buff_head *'
 static inline struct sk_buff *__qdisc_dequeue_head(struct Qdisc *sch,
                               ^
/root/stack/kernel/ipcps/shim-eth-utils.c:88:25: error: too few arguments to function '__qdisc_dequeue_head'
   struct sk_buff *skb = __qdisc_dequeue_head(&qdisc->q);
                         ^
In file included from include/net/pkt_sched.h:7:0,
                 from /root/stack/kernel/ipcps/shim-eth-utils.c:24:
include/net/sch_generic.h:607:31: note: declared here
 static inline struct sk_buff *__qdisc_dequeue_head(struct Qdisc *sch,
                               ^
/root/stack/kernel/ipcps/shim-eth-utils.c: In function 'shim_eth_qdisc_peek':
/root/stack/kernel/ipcps/shim-eth-utils.c:121:17: error: 'struct sk_buff_head' has no member named 'head'
  return qdisc->q.head;
                 ^
/root/stack/kernel/ipcps/shim-eth-utils.c: In function 'shim_eth_qdisc_init':
/root/stack/kernel/ipcps/shim-eth-utils.c:141:2: error: implicit declaration of function 'qdisc_skb_head_init' [-Werror=implicit-function-declaration]
  qdisc_skb_head_init(&qdisc->q);
  ^
/root/stack/kernel/ipcps/shim-eth-utils.c: In function 'shim_eth_qdisc_reset':
/root/stack/kernel/ipcps/shim-eth-utils.c:157:22: warning: passing argument 1 of '__qdisc_reset_queue' from incompatible pointer type
  __qdisc_reset_queue(&qdisc->q);
                      ^
In file included from include/net/pkt_sched.h:7:0,
                 from /root/stack/kernel/ipcps/shim-eth-utils.c:24:
include/net/sch_generic.h:695:20: note: expected 'struct Qdisc *' but argument is of type 'struct sk_buff_head *'
 static inline void __qdisc_reset_queue(struct Qdisc *sch,
                    ^
/root/stack/kernel/ipcps/shim-eth-utils.c:157:2: error: too few arguments to function '__qdisc_reset_queue'
  __qdisc_reset_queue(&qdisc->q);
  ^
In file included from include/net/pkt_sched.h:7:0,
                 from /root/stack/kernel/ipcps/shim-eth-utils.c:24:
include/net/sch_generic.h:695:20: note: declared here
 static inline void __qdisc_reset_queue(struct Qdisc *sch,
                    ^
/root/stack/kernel/ipcps/shim-eth-utils.c: At top level:
/root/stack/kernel/ipcps/shim-eth-utils.c:166:2: warning: initialization from incompatible pointer type
  .enqueue   = shim_eth_qdisc_enqueue,
  ^
/root/stack/kernel/ipcps/shim-eth-utils.c:166:2: warning: (near initialization for 'shim_eth_qdisc_ops.enqueue')
/root/stack/kernel/ipcps/shim-eth-utils.c: In function 'shim_eth_qdisc_peek':
/root/stack/kernel/ipcps/shim-eth-utils.c:122:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^

edugrasa · 2017-09-18T12:28:58Z

v4.6 doesn't have the problem Miguel reported, but the changes to the qdisc API were not there yet. Therefore 4.9 is the minimum version required I'm afraid...

vmaffione · 2017-09-18T15:39:25Z

Ok I see the problem.
A quick hack would be to use linux versions to decide wether to compile IFF_PHONY_HEADROOM and the missing qdisc parameter

#include <linux/version.h>

#if LINUX_VERSION_CODE <= KERNEL_VERSION(4,6,0)
// don't set IFF_PHONY_HEADROOM bit
#else
// set it
#endif

we can upgrade this to kernel feature probing later (look for probe_kernel_features here https://pastebin.com/19wP7Y2E).

edugrasa · 2017-09-18T16:40:04Z

Ok, will try the quick hack, should be ready tomorrow morning

edugrasa · 2017-09-19T12:22:36Z

@miguelpdl @vmaffione
I've prepared a PR with kernel support 4.1 - 4.9 and updated build scripts (just using configure and make)
@miguelpdl could you validate it works?

#1126

miguelpdl · 2017-09-26T16:19:31Z

Back on this item I can confirm that the build works on Debian 8 environment.

And in a round about way it builds on Ubuntu 16.04 too (v4.4 kernel). Most things seems to run fine until an error at the end.

INSTALL /home/kodacall/src/irati/stack/plugins/red/red-plugin.ko
At main.c:222:
- SSL error:02001002:system library:fopen:No such file or directory: bss_file.c:175
- SSL error:2006D080:BIO routines:BIO_new_file:no such file: bss_file.c:178
sign-file: certs/signing_key.pem: No such file or directory
  DEPMOD  4.4.0-93-generic

It's an error so as far as I can tell it does not break the build, but it certainly doesn't look right.

In looking at this issue it appears to be caused by a new and stricter method of kernel module signing on Ubuntu 16.04. More can be read here about Ubuntu 16.04, make install leads to ssl error.

Does this mean a new issue should be raised against the IRATI Stack, in regards to DKMS installation being the recommended way on Ubuntu ?

miguelpdl · 2017-09-26T16:35:12Z

Finally built the IRATI Stack on a GitLab Runner successfully

But there's a few notes to write up before I can request a merge.

miguelpdl · 2017-10-03T08:29:11Z

Another comment here that might be worth raising as an feature request (i.e. Issue).

The current process to build the IRATI Stack is

sudo ./configure --prefix ~/install
sudo make install

But shouldn't the process be

sudo ./configure --prefix ~/install
sudo make
sudo make install

Now I (think) I know why the build is done like this. The configure actually builds and installs librina so that when it comes to rinad it has librina in place. But is there another way completely this neatly.

edugrasa · 2017-10-19T12:30:33Z

Fixed by #1137

miguelpdl · 2017-10-25T15:40:55Z

I've re-opened this issue as the task as first laid out by the ticket "Add Stack build to Travis CI" was not really complete with the last commit.

The last PR really just ensured that the IRATI Stack would build on the Gitlab Runner, running off the mirrored repo for the stack over in Gitalb world.

That is not an ideal scenario as the mirroring can take between 20 minutes to 18 hours to cross over to the Gitlab side, so the real task here was to get the Travis CI service building the stack.

After much trial and error, the Travis CI build service now builds the Stack. Some of the issues encountered included:

Having to build and load a from source version of libprotobuf9 v2.6.1
Having to set the Environment Variables PKG_CONFIG_PATH and LD_LIBRARY_PATH as part of the build variables
Having to remove the dependency on version 0.2.0 in the configuration of rinad and rina-tools. The system still checks that librina is there, but it does not check for a specific version. The reason, this check is not done, is due to the fact that librina uses git-version-gen to create it's PACKAGE_VERSION, however in the Travis CI world the $VERSION that generated by this system is not passed as a global environment variable and thus the VERSION of librina gets set to UNKNOWN and therefore configure just bombs out when checking for librina.
As stated with this change, the configure script still looks for librina but no version is specified.

Now I've merged in the latest commits from the arcfire branch, the build passes again so will created a new merge request and hopefully we can truly close out this ticket.

miguelpdl · 2017-10-25T16:12:36Z

Also of note the build over on Gitlab still passes too with these changes.

miguelpdl added the testing label Jul 18, 2017

miguelpdl added this to the arcfire-1.3 milestone Jul 18, 2017

miguelpdl self-assigned this Jul 18, 2017

miguelpdl pushed a commit that referenced this issue Jul 18, 2017

Adding version of Travis CI YML as per Issue #1109

e0aa1c6

miguelpdl pushed a commit that referenced this issue Jul 18, 2017

Removing wpa-supplicant install for the moment. Issue #1109 .

b7f64ef

edugrasa modified the milestones: arcfire-1.3, arcfire-1.4 Jul 28, 2017

edugrasa closed this as completed Oct 19, 2017

miguelpdl reopened this Oct 25, 2017

miguelpdl mentioned this issue Oct 25, 2017

1109 travis ci build #1158

Merged

miguelpdl mentioned this issue Oct 31, 2017

Enable librina and rinad tests as part of the continuous integration build process #1159

Closed

edugrasa closed this as completed Oct 31, 2017

Add Stack build to Travis CI #1109

Add Stack build to Travis CI #1109

Comments

miguelpdl commented Jul 18, 2017

miguelpdl commented Jul 19, 2017

edugrasa commented Jul 19, 2017

miguelpdl commented Jul 19, 2017

dstaesse commented Jul 31, 2017

msune commented Sep 8, 2017

miguelpdl commented Sep 12, 2017

miguelpdl commented Sep 12, 2017

miguelpdl commented Sep 12, 2017 • edited Loading

edugrasa commented Sep 12, 2017

miguelpdl commented Sep 12, 2017

edugrasa commented Sep 12, 2017

dstaesse commented Sep 12, 2017

edugrasa commented Sep 12, 2017

miguelpdl commented Sep 12, 2017

msune commented Sep 12, 2017

sandervrijders commented Sep 12, 2017 • edited Loading

msune commented Sep 12, 2017

sandervrijders commented Sep 12, 2017

msune commented Sep 12, 2017

sandervrijders commented Sep 12, 2017

vmaffione commented Sep 12, 2017 via email • edited Loading

edugrasa commented Sep 12, 2017

vmaffione commented Sep 12, 2017

edugrasa commented Sep 12, 2017

miguelpdl commented Sep 15, 2017

miguelpdl commented Sep 15, 2017

miguelpdl commented Sep 15, 2017

vmaffione commented Sep 15, 2017 via email • edited Loading

miguelpdl commented Sep 18, 2017

edugrasa commented Sep 18, 2017 • edited Loading

edugrasa commented Sep 18, 2017

vmaffione commented Sep 18, 2017

edugrasa commented Sep 18, 2017

edugrasa commented Sep 19, 2017

miguelpdl commented Sep 26, 2017 • edited Loading

miguelpdl commented Sep 26, 2017

miguelpdl commented Oct 3, 2017 • edited Loading

edugrasa commented Oct 19, 2017

miguelpdl commented Oct 25, 2017 • edited Loading

miguelpdl commented Oct 25, 2017

miguelpdl commented Sep 12, 2017 •

edited

Loading

sandervrijders commented Sep 12, 2017 •

edited

Loading

vmaffione commented Sep 12, 2017 via email •

edited

Loading

vmaffione commented Sep 15, 2017 via email •

edited

Loading

edugrasa commented Sep 18, 2017 •

edited

Loading

miguelpdl commented Sep 26, 2017 •

edited

Loading

miguelpdl commented Oct 3, 2017 •

edited

Loading

miguelpdl commented Oct 25, 2017 •

edited

Loading