Istio ingress: duplicate listener 0.0.0.0_8081 found #10160

jasonnance · 2020-11-19T15:45:12Z

/area networking

What version of Knative?

0.19.x

Expected Behavior

Istio ingress gateway with provided knative-istio-controller configuration exposes Knative services consistently.

Actual Behavior

After a Knative service is added to the knative-local-gateway, the istio-ingressgateway pod reports the following error, and the ingress never comes online:

warning	envoy config	gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8081: duplicate listener 0.0.0.0_8081 found
2

Istio listener config:

$ istioctl proxy-config listeners istio-ingressgateway-xxx.istio-system
ADDRESS PORT  MATCH DESTINATION
0.0.0.0 8081  ALL   Route: http.80
0.0.0.0 15021 ALL   Non-HTTP/Non-TCP
0.0.0.0 15090 ALL   Non-HTTP/Non-TCP

This is running on a "toy" dev cluster which gets spun up and torn down each day and has all state managed using GitOps via Flux. Twice now, after tweaking a bunch of random stuff (ports, deleting/recreating gateways, etc), I've gotten the ingress into a working state, but when the cluster comes back up the following day with the same config, it's broken again.

When it works, the listener config looks like this:

$ istioctl proxy-config listeners istio-ingressgateway-86f88b6f6-plp4p.istio-system
ADDRESS PORT  MATCH DESTINATION
0.0.0.0 8080  ALL   Route: http.80
0.0.0.0 8081  ALL   Route: http.8081
0.0.0.0 15021 ALL   Non-HTTP/Non-TCP
0.0.0.0 15090 ALL   Non-HTTP/Non-TCP

As best I can tell, Istio is incorrectly merging the two gateways (default istio-ingressgateway and knative-local-gateway), but I'm not familiar enough with Istio/Knative Serving or their intersection to understand whether this is a problem with Knative's use of Istio or an Istio bug.

Steps to Reproduce the Problem

k8s 1.17
Istio 1.7.4

knative-istio-controller.yaml (relevant sections only, should be all default):

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: knative-ingress-gateway
  namespace: knative-serving
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cluster-local-gateway
  namespace: knative-serving
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  selector:
    istio: cluster-local-gateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: knative-local-gateway
  namespace: knative-serving
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 8081
        name: http
        protocol: HTTP
      hosts:
        - "*"

---
apiVersion: v1
kind: Service
metadata:
  name: knative-local-gateway
  namespace: istio-system
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  type: ClusterIP
  selector:
    istio: ingressgateway
  ports:
    - name: http2
      port: 80
      targetPort: 8081

Knative service:

---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: kinesis-seldon-adapter
  namespace: polyaxon
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
        - image: ko://my-module
          imagePullPolicy: Always
          ports:
            - containerPort: 8080

Gist with partial Istio proxy config dump showing the duplicate listeners in the failing state and the correct listeners in the working state (again, both states stemming from the same config shown above): https://gist.github.com/jasonnance/885e9772370ac6e46924844646597943

This is part of a more complex workflow involving Knative Eventing, so let me know if I've left out anything relevant.

Thanks!

The text was updated successfully, but these errors were encountered:

vagababov · 2020-11-19T16:58:17Z

/cc @nak3 @JRBANCEL

JRBANCEL · 2020-11-19T20:49:48Z

Weird, the config clearly shows that 0.0.0.0_8081 is coming both from http.80 and http.8081.
Only http.8081 is expected.

What does the k8s service look like? Istio will use the targetPort defined for 80 on Envoy.

kubectl get svc -n istio-system  istio-ingressgateway -o yaml

jasonnance · 2020-11-19T21:04:23Z

k8s service (targetPort appears to be 8080):

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"istio-ingressgateway","install.operator.istio.io/owning-resource":"istio-operator","install.operator.istio.io/owning-resource-namespace":"istio-operator","istio":"ingressgateway","istio.io/rev":"default","operator.istio.io/component":"IngressGateways","operator.istio.io/managed":"Reconcile","operator.istio.io/version":"1.7.4","release":"istio"},"name":"istio-ingressgateway","namespace":"istio-system"},"spec":{"ports":[{"name":"status-port","port":15021,"targetPort":15021},{"name":"http2","port":80,"targetPort":8080},{"name":"https","port":443,"targetPort":8443},{"name":"tls","port":15443,"targetPort":15443}],"selector":{"app":"istio-ingressgateway","istio":"ingressgateway"},"type":"LoadBalancer"}}
  creationTimestamp: "2020-11-19T13:37:49Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app: istio-ingressgateway
    install.operator.istio.io/owning-resource: istio-operator
    install.operator.istio.io/owning-resource-namespace: istio-operator
    istio: ingressgateway
    istio.io/rev: default
    operator.istio.io/component: IngressGateways
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.7.4
    release: istio
  name: istio-ingressgateway
  namespace: istio-system
  resourceVersion: "563142"
  selfLink: /api/v1/namespaces/istio-system/services/istio-ingressgateway
  uid: 1943515f-83e7-400f-bab3-16d9a2c095df
spec:
  clusterIP: x.x.x.x
  externalTrafficPolicy: Cluster
  ports:
  - name: status-port
    nodePort: 32326
    port: 15021
    protocol: TCP
    targetPort: 15021
  - name: http2
    nodePort: 30029
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 31002
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: tls
    nodePort: 30347
    port: 15443
    protocol: TCP
    targetPort: 15443
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - hostname: xxx-xxx.us-east-1.elb.amazonaws.com

Here's my Istio operator config as well -- realized that might be relevant (although it's also mostly default):

# Adapted from https://knative.dev/docs/install/installing-istio/#installing-istio-without-sidecar-injection
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-operator
  namespace: istio-operator
spec:
  values:
    global:
      proxy:
        autoInject: disabled
      useMCP: false
      # The third-party-jwt is not enabled on all k8s.
      # See: https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens
      jwtPolicy: first-party-jwt
      tracer:
        zipkin:
          address: jaeger-collector.observability.svc.cluster.local:9411

    pilot:
      traceSampling: 1.0

  addonComponents:
    pilot:
      enabled: true
    prometheus:
      enabled: false

  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
      - name: cluster-local-gateway
        enabled: true
        label:
          istio: cluster-local-gateway
          app: cluster-local-gateway
        k8s:
          service:
            type: ClusterIP
            ports:
            - port: 15020
              name: status-port
            - port: 80
              targetPort: 8080
              name: http2
            - port: 443
              targetPort: 8443
              name: https

Shashankft9 · 2021-01-04T14:35:26Z

Getting the same error while installing v0.19.0:

[external/envoy/source/common/config/grpc_subscription_impl.cc:101] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8081: duplicate listener 0.0.0.0_8081 found

istioctl proxy-config listeners istio-ingressgateway-bgkqm.istio-system
ADDRESS     PORT      TYPE
0.0.0.0     8081      HTTP
0.0.0.0     15090     HTTP
0.0.0.0     15021     HTTP

Because of this, rest of the ingress traffic also stops working since I have made the gateways on port 80. Without knative-serving, I have an entry there with 80 instead of 8081. 80 because unlike @jasonnance I have targetPort and port both set to be 80

JRBANCEL · 2021-01-05T21:37:06Z

@howardjohn Any idea how to debug this?

howardjohn · 2021-01-05T21:39:20Z

I think the issue is you have a Service for port 80 -> target 8081, a Gateway for 80 (which maps to 0.0.0.0_8081 due to targetPort) and a Gateway for 8081 (which maps to 0.0.0.0_8081 due to targetPort). Istio doesn't like this I guess.

I recall the same issue before, not sure if we have solved it already in a newer version or if its an open issue in istio/istio.

howardjohn · 2021-01-05T21:40:16Z

To clarify, its probably both a user error (not sure why you would want the above config - both Gateways should be port 80 probably) and an Istio bug (if user does it, we need to handle it gracefully)

JRBANCEL · 2021-01-05T21:43:51Z

The ingress Gateway listens to 8080 and the matching k8s svc maps 80 -> 8080 (default Istio setup, nothing specific to Knative).

Then, we have a (Knative) Gateway listen on 8081 and a matching k8s svc mapping 80 -> 8080.

This should work and does work (that's what runs in the Knative E2E tests).

I am not sure what is the issue in this particular scenario though.

howardjohn · 2021-01-05T21:48:14Z

In the original issue:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: knative-local-gateway
  namespace: knative-serving
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 8081
        name: http
        protocol: HTTP
      hosts:
        - "*"
---
apiVersion: v1
kind: Service
metadata:
  name: knative-local-gateway
  namespace: istio-system
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  type: ClusterIP
  selector:
    istio: ingressgateway
  ports:
    - name: http2
      port: 80
      targetPort: 8081
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: knative-ingress-gateway
  namespace: knative-serving
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

These 3 are enough to trigger it, then you have two Gateways which both map to port 8081. I don't think its standard knative setup, which is likely why its not a more common issue.

JRBANCEL · 2021-01-05T21:54:59Z

then you have two Gateways which both map to port 8081

Why though?

knative-local-gateway wants to listen on 8081 and the matching service targetPort for 80 is also 8081, so Envoy actually listens on 8081.

knative-ingress-gateway wants to listen on 80 but the matching service (default Istio setup) istio-ingressgateway has a targetPort of 8080 for port 80 so Envoy actually listens on 8080.

Am I missing something? This is the new Knative setup and it should work.

howardjohn · 2021-01-05T22:03:18Z

Ah, I think I missed part of it. The confusion is matching service I think. The mapping of knative-local-gateway -> 80/8081 service and knative-ingress-gateway -> 80/8080 service is not quite right.

There is a single istio-ingressgateway deployment, which has two services. one of them maps port 80 to 8080 and the other maps port 80 to 8081. So far, no problem.

Then we create a Gateway on port 80, which sets up listeners on port 8080 and 8081 (Maybe? I actually doubt this works, need to verify), all good

Then we create a Gateway on port 80801, which sets up another listener on port 8081 (it shouldn't, but the logic is wrong and doesn't merge), which breaks

JRBANCEL · 2021-01-05T22:17:42Z

Then we create a Gateway on port 80, which sets up listeners on port 8080 and 8081 (Maybe? I actually doubt this works, need to verify), all good

Hmm, maybe this is another instance where Istio is not deterministic and picks just one port?

This is worrying. This has been running fine in our E2E tests for the past month and we don't see this issue (both 1.7.x and 1.8.x)

howardjohn · 2021-01-05T22:21:24Z

@JRBANCEL do you have the same setup with multiple services with different port setups in your e2e tests? I was under the impression the original issue has some customization not in the standard knative setup that was triggering it

JRBANCEL · 2021-01-05T22:23:13Z

We install Istio with this profile: https://github.com/knative-sandbox/net-istio/blob/master/third_party/istio-stable/istio-ci-no-mesh.yaml

And these are the additional Gateway and Service objects:
https://github.com/knative-sandbox/net-istio/blob/master/config/202-gateway.yaml
https://github.com/knative-sandbox/net-istio/blob/master/config/203-local-gateway.yaml

howardjohn · 2021-01-05T22:25:54Z

Huh.. if you are not seeing issues then I may be mistaken. We will need to investigate this more on the Istio side

howardjohn · 2021-01-05T22:27:28Z

istio/istio#29643
istio/istio#29291

are similar issue on Istio tracker

JRBANCEL · 2021-01-05T22:42:21Z

Then we create a Gateway on port 80, which sets up listeners on port 8080 and 8081 (Maybe? I actually doubt this works, need to verify), all good

Can you check the implementation when you have a second because that is not what I am seeing.

Fresh Istio 1.8.1 install with the profile linked above:

ADDRESS PORT  MATCH DESTINATION
0.0.0.0 15021 ALL   Inline Route: /healthz/ready*
0.0.0.0 15090 ALL   Inline Route: /stats/prometheus*

After installing the ingress Gateway:

$ ko apply -f config/202-gateway.yaml 
gateway.networking.istio.io/knative-ingress-gateway created

ADDRESS PORT  MATCH DESTINATION
0.0.0.0 8080  ALL   Route: http.80
0.0.0.0 15021 ALL   Inline Route: /healthz/ready*
0.0.0.0 15090 ALL   Inline Route: /stats/prometheus*

After installing the local Gateway and Service:

$ ko apply -f config/203-local-gateway.yaml                                                                      
gateway.networking.istio.io/knative-local-gateway created
service/knative-local-gateway created

ADDRESS PORT  MATCH DESTINATION
0.0.0.0 8080  ALL   Route: http.80
0.0.0.0 8081  ALL   Route: http.8081
0.0.0.0 15021 ALL   Inline Route: /healthz/ready*
0.0.0.0 15090 ALL   Inline Route: /stats/prometheus*

Even when I delete the local Gateway and leave the matching Service, there is nothing on 8081:

ADDRESS PORT  MATCH DESTINATION
0.0.0.0 8080  ALL   Route: http.80
0.0.0.0 15021 ALL   Inline Route: /healthz/ready*
0.0.0.0 15090 ALL   Inline Route: /stats/prometheus*

Shashankft9 · 2021-01-06T10:17:47Z

For me, with all the knative components installed, it gives:

ADDRESS     PORT      TYPE
0.0.0.0     8081      HTTP
0.0.0.0     15090     HTTP
0.0.0.0     15021     HTTP

the above stays when I delete the knative-local-gateway, and only when I delete the knative-local-gateway svc, it goes back to:

ADDRESS     PORT      TYPE
0.0.0.0     80        HTTP
0.0.0.0     15090     HTTP
0.0.0.0     15021     HTTP

mattmoor · 2021-01-06T19:36:53Z

/assign @JRBANCEL

Moving this off of the serving triage query, we should move these discussions to net-istio.

mjgallag · 2021-02-03T23:43:07Z

Not sure it's helpful to root causing this but I just wanted to confirm that it does appear to be a timing issue (not deterministic). I worked around the problem by installing and waiting for istio to be ready first.

Shashankft9 · 2021-02-18T12:57:02Z

@mjgallag hey just to check, what happens if you happen to delete istio and reinstall it again while knative-serving is running?
I deployed istio on a new cluster, then knative-serving and then for some reason had to reinstall istio, and that is when I got this issue again.
Just checking if this issue is reproducible this way?

mjgallag · 2021-02-19T02:35:53Z

@Shashankft9 I didn't get this issue, but it did cause knative issues. A non knative service I have routed thru istio worked fine again right after I reinstalled istio. As for my knative services, they returned no healthy upstream even though I checked underlying revision pod containers and they all seemed fine. In fact, I couldn't find anything wrong with knative services, routes & revisions. I also looked at the istio gateways & virtual services created by knative for my services plus the envoy listeners & routes they create and they all looked correct too. I was able to deploy new revisions successfully but still got no healthy upstream. I had to delete and recreate my services for them to work again.

JRBANCEL · 2021-02-25T21:15:47Z

I reported a minimal repro of this issue to Istio: istio/istio#31084

ZhuangYuZY · 2021-03-02T07:02:07Z

It also happens in my env today when update istio. It impacts http endpoint access from external. As LB forward http 80 traffic to istio gateway port 8080. But 8080 missing from istio listener. It seems like timing issue in istio. I workaround the issue by remove knative-local-gateway service in istio-system namespace.

Before change:

( istioctl proxy-config listener istio-ingressgateway-5b69f56d57-nl4x6.istio-system
ADDRESS PORT  MATCH                                                             DESTINATION
0.0.0.0 8081  ALL                                                               Route: http.80          //8080 missing
0.0.0.0 8089  ALL                                                               Route: http.8089    //before I change port to 8089 in knative-local-gateway to workaround the issue.

k delete svc -n istio-system knative-local-gateway

( istioctl proxy-config listener istio-ingressgateway-5b69f56d57-nl4x6.istio-system
ADDRESS PORT  MATCH                                                             DESTINATION
0.0.0.0 8080  ALL                                                               Route: http.80        //8080 listener back, which define in istio-ingressgateway service by default in istio-system namespace.
0.0.0.0 8089  ALL                                                               Route: http.8089

k delete gateway -n knative-serving knative-local-gateway
gateway.networking.istio.io "knative-local-gateway" deleted
knative-local-gateway seem recreated.

(istioctl proxy-config listener istio-ingressgateway-5b69f56d57-nl4x6.istio-system
ADDRESS PORT  MATCH                                                             DESTINATION
0.0.0.0 8080  ALL                                                               Route: http.80
0.0.0.0 8081  ALL                                                               Route: http.8081    //8081 back as normall, we removed knative-local-gateway, and recreate it as default.

create knative-local-gateway service, istio listener still correct.

(dev-serving-s01:default)root@coligo-test:~# istioctl proxy-config listener istio-ingressgateway-5b69f56d57-nl4x6.istio-system
ADDRESS PORT  MATCH                                                             DESTINATION
0.0.0.0 8080  ALL                                                               Route: http.80
0.0.0.0 8081  ALL                                                               Route: http.8081

howardjohn · 2021-03-16T05:22:04Z

This should be fixed. in istio by istio/istio@ae8b0e2

evankanderson · 2021-03-19T19:36:34Z

Looks like this may be fixed in Istio.

/close

knative-prow-robot · 2021-03-19T19:36:39Z

@evankanderson: Closing this issue.

In response to this:

Looks like this may be fixed in Istio.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nak3 · 2021-04-12T06:58:44Z

It seems istio/istio@ae8b0e2 did not fix the issue.
We still need to track istio/istio#31084

nak3 · 2021-06-03T06:10:37Z

Quick update, istio/istio#33021 should fix the issue.

dprotaso · 2021-06-11T19:29:38Z

Related: knative-extensions/net-istio#636

We'll need to pull in the next 1.9.x patch release when it's out

dprotaso · 2021-07-05T22:53:49Z

We're pulling in the latest patches now - knative-extensions/net-istio#699

houshym · 2021-07-13T19:28:23Z

I can confirm this issue exists in istio 1.9.6. kubernetes version 1.20.7

dprotaso · 2021-07-13T23:01:08Z

@houshym do you have the following ~~annotation~~ label on your gateway?

https://github.com/knative-sandbox/net-istio/pull/636/files

 experimental.istio.io/disable-gateway-port-translation: "true"

houshym · 2021-07-13T23:03:34Z

@dprotaso I don't have. should I add it?

dprotaso · 2021-07-13T23:09:05Z

According to that PR having that label on the gateway (with istio 1.9.6) should activate that functionality which supposedly fixes the behaviour. Hence why I was saying that should fix the issue for Knative's installation of Istio.

dprotaso · 2021-07-13T23:13:50Z

Let me know if it doesn't - then maybe we should reach out to some Istio folks.

I think for now we can close this issue - ~~I'm going to move it to the net-istio repo as this is specific to that~~

ok ... I can't transfer to another org :(

jasonnance added the kind/bug Categorizes issue or PR as related to a bug. label Nov 19, 2020

knative-prow-robot added the area/networking label Nov 19, 2020

knative-prow-robot assigned JRBANCEL Jan 6, 2021

JRBANCEL mentioned this issue Feb 25, 2021

Unexpected listeners configuration when multiple k8s svc match a Gateway istio/istio#31084

Closed

bianpengyuan mentioned this issue Mar 12, 2021

Update gateway port to service port before gateway merging. istio/istio#31386

Closed

knative-prow-robot closed this as completed Mar 19, 2021

dprotaso reopened this Apr 12, 2021

dprotaso added this to the Ice Box milestone Jun 11, 2021

lsuanet mentioned this issue Jun 17, 2021

404 on all routes using simple example kserve/kserve#1665

Closed

dprotaso modified the milestones: Ice Box, 0.25.x Jul 5, 2021

amybachir mentioned this issue Jul 12, 2021

duplicate listener 0.0.0.0_8081 found argoflow/argoflow-aws#171

Open

dprotaso closed this as completed Jul 13, 2021

kimwnasptd mentioned this issue Dec 6, 2021

Istio and Knative problematic Gateway ports mapping in KF 1.4 kubeflow/manifests#2082

Closed

nak3 mentioned this issue Jul 10, 2023

Drop workaround patch for Istio #14156

Merged

Istio ingress: duplicate listener 0.0.0.0_8081 found #10160

Istio ingress: duplicate listener 0.0.0.0_8081 found #10160

Comments

jasonnance commented Nov 19, 2020

What version of Knative?

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

vagababov commented Nov 19, 2020

JRBANCEL commented Nov 19, 2020 • edited Loading

jasonnance commented Nov 19, 2020

Shashankft9 commented Jan 4, 2021

JRBANCEL commented Jan 5, 2021

howardjohn commented Jan 5, 2021

howardjohn commented Jan 5, 2021

JRBANCEL commented Jan 5, 2021 • edited Loading

howardjohn commented Jan 5, 2021

JRBANCEL commented Jan 5, 2021

howardjohn commented Jan 5, 2021

JRBANCEL commented Jan 5, 2021

howardjohn commented Jan 5, 2021

JRBANCEL commented Jan 5, 2021

howardjohn commented Jan 5, 2021

howardjohn commented Jan 5, 2021

JRBANCEL commented Jan 5, 2021

Shashankft9 commented Jan 6, 2021

mattmoor commented Jan 6, 2021

mjgallag commented Feb 3, 2021

Shashankft9 commented Feb 18, 2021

mjgallag commented Feb 19, 2021

JRBANCEL commented Feb 25, 2021 • edited Loading

ZhuangYuZY commented Mar 2, 2021

howardjohn commented Mar 16, 2021

evankanderson commented Mar 19, 2021

knative-prow-robot commented Mar 19, 2021

nak3 commented Apr 12, 2021

nak3 commented Jun 3, 2021

dprotaso commented Jun 11, 2021

dprotaso commented Jul 5, 2021

houshym commented Jul 13, 2021

dprotaso commented Jul 13, 2021 • edited Loading

houshym commented Jul 13, 2021

dprotaso commented Jul 13, 2021 • edited Loading

dprotaso commented Jul 13, 2021 • edited Loading

JRBANCEL commented Nov 19, 2020 •

edited

Loading

JRBANCEL commented Jan 5, 2021 •

edited

Loading

JRBANCEL commented Feb 25, 2021 •

edited

Loading

dprotaso commented Jul 13, 2021 •

edited

Loading

dprotaso commented Jul 13, 2021 •

edited

Loading

dprotaso commented Jul 13, 2021 •

edited

Loading