Netpol test suite randomly fails when run in Kind and with tunnel type set to Geneve #897
Comments
abhiraut
added
kind/bug
Set default tunnel type to Geneve for Antrea instead of VXLAN. When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
Instead of running whole suite for every alpha feature, it was discussed to run the suite by enabling all alpha features. This PR handles alpha features on Antrea Agent. As there is ongoing geneve tunnel issue, Issue antrea-io#897 with ClusterNetworkPolicy alpha feature on controller, I did not club alpha features on agent and controller. Similarly, Traceflow feature e2e test could be modified, when both agent and controller alpha features are clubbed together. This PR is useful to turn on e2e tests for flow exporter feature PR antrea-io#825. Modified e2e documentation to run alpha feature specific tests locally. Issue antrea-io#893
Instead of running whole suite for every alpha feature, it was discussed to run the suite by enabling all alpha features. This PR handles alpha features on Antrea Agent. As there is ongoing geneve tunnel issue, Issue antrea-io#897 with ClusterNetworkPolicy alpha feature on controller, I did not club alpha features on agent and controller. Similarly, Traceflow feature e2e test could be modified, when both agent and controller alpha features are clubbed together. This PR is useful to turn on e2e tests for flow exporter feature PR antrea-io#825. Modified e2e documentation to run alpha feature specific tests locally. Issue antrea-io#893
|
Changing networkPolicyDelay to 20s helps in passing the test for geneve tunnel. I ran network policy realization benchmark test with geneve and vxlan tunnel to see difference in realization times: With vxlan tunnel: With geneve tunnel: There is 10x difference in realization times for Geneve when compared to VXLAN. As a fix, we could check with wait/poll if network policy is realized on at least one of the agents/required agent using antctl rather than sleeping for constant time. Please suggest if you could think of an alternative. |
|
It's more like a 4x difference, but I agree it could explain the issue under certain circumstances. Before thinking about a fix, we need to find an explanation for this if we can:
The communication channels you mention do not involve the inter-Node tunnels, everything should happen over the host network directly, which is why this is a bit confusing... |
|
Yes its not 10x.. don't know why I said that (I was looking at other numbers in parallel). |
|
Checked logs when running netpol tests with timestamps. I see that rules being realized quickly on agent and syncing happens in milliseconds. In addition, checked prometheus metrics (antrea_agent_ovs_flow_ops_latency) at agent corresponding to AddFlowsInBundle. Add operations are finishing in milliseconds. Following is the failed case. Note the difference in time zones: Agent log: Complete agent log: BenchmarkRealizeNetworkPolicy/RealizeNetworkPolicy5000 is a false positive. After running multiple times, observed that the nanosecond/operation metrics has quite a bit of range for both vxlan and geneve tunnels. In addition, I do not see any rules being realized on the agent. Ignoring this for now. From above time logs when running failed netpol tests, I feel there may be more to this issue and the cause does not seem to be network policy not being realized on agent. Confirmed from code that time elapsed syncing rule/reconciling at agent involves the time installing of flows for the rule. |
|
Investigated this further. Issue seems to be dataplane flow cache. Even though the rule is installed, cached flow in dataplane is processing the old rule/action, which is causing random failures with geneve tunnel. I observed that communication is failing for remote pods for whom the rules are changed from previous testcase. If the rule doesn't change, then communication is fine. With my limited OVS debugging skills, I tried a bit but could not figure out the exact reason why it is happening only for geneve tunnel and not vxlan tunnel :) |
|
Thanks for looking into this. Let me see if I can reproduce outside of Antrea with a standalone OVS switch, since you are probably busy with the Flow Export stuff. If I can, we can let the OVS team know. |
Sounds good. Thanks. |
probably let ovs team take a look? we anyway have set the netpol tests to run in vxlan mode for now. maybe add a note about it in some sort of known issues doc ? but if it needs a workaround for testing other alpha features (as suggested by @srikartati ) then maybe we can alter the param for testing before ovs team fixes it. |
|
Good news. I think I was able to reproduce the issue outside of Antrea, with a vanilla OVS deployment. I will summarize the steps and reach out to the OVS team. |
When running NetworkPolicy tests in Kind, we "force" the tunnel type to VXLAN as we observe some failures with Geneve (see antrea-io#897). We also change the Github CI jobs so that CNP tests are run in their own job. This is to address the issue above, and is also a partial implementation of antrea-io#893.
|
It seems a patch was submitted upstream: https://www.mail-archive.com/ovs-dev@openvswitch.org/msg49689.html. We can pick it up in Antrea after it's merged. |
On userspace datapath, geneve option flag FLOW_TNL_F_UDPIF should only
be set when the geneve option is available. However, currently, we always
set FLOW_TNL_F_UDPIF in the metadata flag, and that may result in megaflow
revalidation issue when the geneve option length is zero due to the datapath
flow key mis-match. That is the original flow key always has FLOW_TNL_F_UDPIF
set, while the regenerated flow key during revalidation process only has
FLOW_TNL_F_UDPIF set if geneve option length is zero. For reference, check
tun_metadata_to_geneve_nlattr() to see how the flow key of geneve
tunnel metadatafor are generated from netlink attributes.
The following are the reproducible steps reported by Antonin.
After step 6, OvS reports a warning about failing to update the datapath
flow.
2020-12-15T01:56:21.324Z|00007|dpif(revalidator4)|WARN|netdev@ovs-netdev:
failed to put[modify] (No such file or directory)
ufid:d252fe62-5b2a-44e6-846a-190328190e09 skb_priority(0/0),
tunnel(tun_id=0x0,src=192.168.77.1,dst=192.168.77.2,ttl=64/0,tp_src=57391/0,
tp_dst=6081/0,flags(-df-csum+key)),skb_mark(0/0),ct_state(0x21/0x21),
ct_zone(0xfff0/0),ct_mark(0/0),ct_label(0/0),ct_tuple4(src=10.10.0.1/0.0.0.0,
dst=10.10.0.2/0.0.0.0,proto=1/0,tp_src=8/0,tp_dst=0/0),recirc_id(0xa),
dp_hash(0/0),in_port(5),packet_type(ns=0,id=0),
eth(src=1e:9c:f0:fb:18:9c/00:00:00:00:00:00,dst=ee:9f:60:8a:c0:a5/00:00:00:00:00:00),
eth_type(0x0800),ipv4(src=10.10.0.1/0.0.0.0,dst=10.10.0.2,proto=1/0,tos=0/0,
ttl=64/0,frag=no),icmp(type=8/0,code=0/0), actions:drop
Setup:
2 Nodes on the same subnet (192.168.77.101/24 - 192.168.77.102/24)
Step 1 – Creating bridges (run on each Node):
iface="enp0s8"
hwaddr=$(ip link show $iface | grep link/ether | awk '{print $2}')
inet=$(ip addr show $iface | grep "inet " | awk '{ print $2 }')
ovs-vsctl add-br br-phy \
-- set Bridge br-phy datapath_type=netdev \
-- br-set-external-id br-phy bridge-id br-phy \
-- set bridge br-phy fail-mode=standalone \
other_config:hwaddr="$hwaddr"
ovs-vsctl --timeout 10 add-port br-phy "$iface"
ip addr add "$inet" dev br-phy
ip link set br-phy up
ip addr flush dev enp0s8 2>/dev/null
ip link set enp0s8 up
iptables -F
ovs-vsctl add-br br-int \
-- set Bridge br-int datapath_type=netdev \
-- set bridge br-phy fail-mode=standalone
Step 2 – Creating netns to for overlay endpoints:
On first Node (192.168.77.101):
ip netns add ns0
ip link add veth0 type veth peer name veth1
ip link set veth0 netns ns0
ovs-vsctl add-port br-int veth1
ip link set dev veth1 up
ip netns exec ns0 ip addr add 10.10.0.1/24 dev veth0
ip netns exec ns0 ip link set dev veth0 up
On second Node (192.168.77.102):
ip netns add ns0
ip link add veth0 type veth peer name veth1
ip link set veth0 netns ns0
ovs-vsctl add-port br-int veth1
ip link set dev veth1 up
ip netns exec ns0 ip addr add 10.10.0.2/24 dev veth0
ip netns exec ns0 ip link set dev veth0 up
Step 3 – Create tunnel (run on each Node):
ovs-vsctl add-port br-int tun0 -- set interface tun0 type=geneve \
ofport_request=100 options:remote_ip=flow options:key=flow
Step 4 – Create the following flows:
On first Node (192.168.77.101):
root@ovs-test-node-1:/home/vagrant# ovs-ofctl dump-flows br-int --no-stats
priority=100,ip actions=resubmit(,10)
priority=0 actions=NORMAL
priority=50 actions=resubmit(,40)
table=10, priority=100,ip actions=ct(table=20,zone=65520)
table=20, priority=200,ct_state=-new+trk,ip actions=resubmit(,30)
table=20, priority=100,ip,nw_dst=10.10.0.2 actions=resubmit(,30)
table=20, priority=0,ip actions=drop
table=30, priority=100,ip actions=ct(commit,table=40,zone=65520)
table=40, priority=100,in_port=veth1 actions=load:0xc0a84d66->NXM_NX_TUN_IPV4_DST[],output:tun0
table=40, priority=100,in_port=tun0 actions=output:veth1
table=40, priority=0 actions=drop
On second Node (192.168.77.102):
root@ovs-test-node-2:/home/vagrant# ovs-ofctl dump-flows br-int --no-stats
priority=100,ip actions=resubmit(,10)
priority=0 actions=NORMAL
priority=50 actions=resubmit(,40)
table=10, priority=100,ip actions=ct(table=20,zone=65520)
table=20, priority=200,ct_state=-new+trk,ip actions=resubmit(,30)
table=20, priority=100,ip,nw_dst=10.10.0.2 actions=resubmit(,30)
table=20, priority=0,ip actions=drop
table=30, priority=100,ip actions=ct(commit,table=40,zone=65520)
table=40, priority=100,in_port=veth1 actions=load:0xc0a84d65->NXM_NX_TUN_IPV4_DST[],output:tun0
table=40, priority=100,in_port=tun0 actions=output:veth1
table=40, priority=0 actions=drop
Step 5 – Check that ping works:
From the first Node: ip netns exec ns0 ping 10.10.0.2
Step 6 – The actual issue:
From the first Node: ip netns exec ns0 ping -c 3 10.10.0.2
From the second Node: ovs-ofctl del-flows br-int 'table=20,ip,nw_dst=10.10.0.2'
From the first Node: ip netns exec ns0 ping -c 3 10.10.0.2
Execute these instructions in order, as soon as the previous one
completes. If you follow these steps, you should see the ping in
the last step succeed. This is not expected because of the deleted
flow. It also does not happen with VXLAN.
Reported-by: Antonin Bas <abas@vmware.com>
Reported-at: antrea-io/antrea#897
Fixes: 6b241d6 ("netdev-vport: Factor-out tunnel Push-pop code into separate module.")
Signed-off-by: Yi-Hung Wei <yihung.wei@gmail.com>
On userspace datapath, geneve option flag FLOW_TNL_F_UDPIF should only
be set when the geneve option is available. However, currently, we always
set FLOW_TNL_F_UDPIF in the metadata flag, and that may result in megaflow
revalidation issue when the geneve option length is zero due to the datapath
flow key mis-match. That is the original flow key always has FLOW_TNL_F_UDPIF
set, while the regenerated flow key during revalidation process only has
FLOW_TNL_F_UDPIF set if geneve option length is zero. For reference, check
tun_metadata_to_geneve_nlattr() to see how the flow key of geneve
tunnel metadatafor are generated from netlink attributes.
The following are the reproducible steps reported by Antonin.
After step 6, OvS reports a warning about failing to update the datapath
flow.
2020-12-15T01:56:21.324Z|00007|dpif(revalidator4)|WARN|netdev@ovs-netdev:
failed to put[modify] (No such file or directory)
ufid:d252fe62-5b2a-44e6-846a-190328190e09 skb_priority(0/0),
tunnel(tun_id=0x0,src=192.168.77.1,dst=192.168.77.2,ttl=64/0,tp_src=57391/0,
tp_dst=6081/0,flags(-df-csum+key)),skb_mark(0/0),ct_state(0x21/0x21),
ct_zone(0xfff0/0),ct_mark(0/0),ct_label(0/0),ct_tuple4(src=10.10.0.1/0.0.0.0,
dst=10.10.0.2/0.0.0.0,proto=1/0,tp_src=8/0,tp_dst=0/0),recirc_id(0xa),
dp_hash(0/0),in_port(5),packet_type(ns=0,id=0),
eth(src=1e:9c:f0:fb:18:9c/00:00:00:00:00:00,dst=ee:9f:60:8a:c0:a5/00:00:00:00:00:00),
eth_type(0x0800),ipv4(src=10.10.0.1/0.0.0.0,dst=10.10.0.2,proto=1/0,tos=0/0,
ttl=64/0,frag=no),icmp(type=8/0,code=0/0), actions:drop
Setup:
2 Nodes on the same subnet (192.168.77.101/24 - 192.168.77.102/24)
Step 1 – Creating bridges (run on each Node):
iface="enp0s8"
hwaddr=$(ip link show $iface | grep link/ether | awk '{print $2}')
inet=$(ip addr show $iface | grep "inet " | awk '{ print $2 }')
ovs-vsctl add-br br-phy \
-- set Bridge br-phy datapath_type=netdev \
-- br-set-external-id br-phy bridge-id br-phy \
-- set bridge br-phy fail-mode=standalone \
other_config:hwaddr="$hwaddr"
ovs-vsctl --timeout 10 add-port br-phy "$iface"
ip addr add "$inet" dev br-phy
ip link set br-phy up
ip addr flush dev enp0s8 2>/dev/null
ip link set enp0s8 up
iptables -F
ovs-vsctl add-br br-int \
-- set Bridge br-int datapath_type=netdev \
-- set bridge br-phy fail-mode=standalone
Step 2 – Creating netns to for overlay endpoints:
On first Node (192.168.77.101):
ip netns add ns0
ip link add veth0 type veth peer name veth1
ip link set veth0 netns ns0
ovs-vsctl add-port br-int veth1
ip link set dev veth1 up
ip netns exec ns0 ip addr add 10.10.0.1/24 dev veth0
ip netns exec ns0 ip link set dev veth0 up
On second Node (192.168.77.102):
ip netns add ns0
ip link add veth0 type veth peer name veth1
ip link set veth0 netns ns0
ovs-vsctl add-port br-int veth1
ip link set dev veth1 up
ip netns exec ns0 ip addr add 10.10.0.2/24 dev veth0
ip netns exec ns0 ip link set dev veth0 up
Step 3 – Create tunnel (run on each Node):
ovs-vsctl add-port br-int tun0 -- set interface tun0 type=geneve \
ofport_request=100 options:remote_ip=flow options:key=flow
Step 4 – Create the following flows:
On first Node (192.168.77.101):
root@ovs-test-node-1:/home/vagrant# ovs-ofctl dump-flows br-int --no-stats
priority=100,ip actions=resubmit(,10)
priority=0 actions=NORMAL
priority=50 actions=resubmit(,40)
table=10, priority=100,ip actions=ct(table=20,zone=65520)
table=20, priority=200,ct_state=-new+trk,ip actions=resubmit(,30)
table=20, priority=100,ip,nw_dst=10.10.0.2 actions=resubmit(,30)
table=20, priority=0,ip actions=drop
table=30, priority=100,ip actions=ct(commit,table=40,zone=65520)
table=40, priority=100,in_port=veth1 actions=load:0xc0a84d66->NXM_NX_TUN_IPV4_DST[],output:tun0
table=40, priority=100,in_port=tun0 actions=output:veth1
table=40, priority=0 actions=drop
On second Node (192.168.77.102):
root@ovs-test-node-2:/home/vagrant# ovs-ofctl dump-flows br-int --no-stats
priority=100,ip actions=resubmit(,10)
priority=0 actions=NORMAL
priority=50 actions=resubmit(,40)
table=10, priority=100,ip actions=ct(table=20,zone=65520)
table=20, priority=200,ct_state=-new+trk,ip actions=resubmit(,30)
table=20, priority=100,ip,nw_dst=10.10.0.2 actions=resubmit(,30)
table=20, priority=0,ip actions=drop
table=30, priority=100,ip actions=ct(commit,table=40,zone=65520)
table=40, priority=100,in_port=veth1 actions=load:0xc0a84d65->NXM_NX_TUN_IPV4_DST[],output:tun0
table=40, priority=100,in_port=tun0 actions=output:veth1
table=40, priority=0 actions=drop
Step 5 – Check that ping works:
From the first Node: ip netns exec ns0 ping 10.10.0.2
Step 6 – The actual issue:
From the first Node: ip netns exec ns0 ping -c 3 10.10.0.2
From the second Node: ovs-ofctl del-flows br-int 'table=20,ip,nw_dst=10.10.0.2'
From the first Node: ip netns exec ns0 ping -c 3 10.10.0.2
Execute these instructions in order, as soon as the previous one
completes. If you follow these steps, you should see the ping in
the last step succeed. This is not expected because of the deleted
flow. It also does not happen with VXLAN.
Reported-by: Antonin Bas <abas@vmware.com>
Reported-at: antrea-io/antrea#897
Fixes: 6b241d6 ("netdev-vport: Factor-out tunnel Push-pop code into separate module.")
Signed-off-by: Yi-Hung Wei <yihung.wei@gmail.com>
Signed-off-by: 0-day Robot <robot@bytheb.org>
|
There is a new upstream patch for OVS which I believe replaces the one mentioned in #897 (comment): https://mail.openvswitch.org/pipermail/ovs-dev/2020-December/378711.html. The patch has not been merged yet. |
This optimization caused FLOW_TNL_F_UDPIF flag not to be used in hash calculation for geneve tunnel when revalidating flows which resulted in different cache hash values and incorrect behaviour. Added test to prevent regression. Reported-at: antrea-io/antrea#897 Signed-off-by: Toms Atteka <cpp.code.lv@gmail.com> Signed-off-by: 0-day Robot <robot@bytheb.org>
This optimization caused FLOW_TNL_F_UDPIF flag not to be used in hash calculation for geneve tunnel when revalidating flows which resulted in different cache hash values and incorrect behaviour. Added test to prevent regression. Reported-at: antrea-io/antrea#897 Signed-off-by: Toms Atteka <cpp.code.lv@gmail.com> Signed-off-by: 0-day Robot <robot@bytheb.org>
This optimization caused FLOW_TNL_F_UDPIF flag not to be used in hash calculation for geneve tunnel when revalidating flows which resulted in different cache hash values and incorrect behaviour. Added test to prevent regression. CC: Jesse Gross <jesse@nicira.com> Fixes: 6728d57 ("dpif-netdev: Translate Geneve options per-flow, not per-packet.") Reported-at: antrea-io/antrea#897 Signed-off-by: Toms Atteka <cpp.code.lv@gmail.com> Signed-off-by: 0-day Robot <robot@bytheb.org>
This optimization caused FLOW_TNL_F_UDPIF flag not to be used in hash calculation for geneve tunnel when revalidating flows which resulted in different cache hash values and incorrect behaviour. Added test to prevent regression. CC: Jesse Gross <jesse@nicira.com> Fixes: 6728d57 ("dpif-netdev: Translate Geneve options per-flow, not per-packet.") Reported-at: antrea-io/antrea#897 Signed-off-by: Toms Atteka <cpp.code.lv@gmail.com> Acked-by: Ansis Atteka <aatteka@ovn.org>
|
The patch has been merged into the master branch of OVS: openvswitch/ovs@cca4014. We should consider applying it to our OVS build. |
This optimization caused FLOW_TNL_F_UDPIF flag not to be used in hash calculation for geneve tunnel when revalidating flows which resulted in different cache hash values and incorrect behaviour. Added test to prevent regression. CC: Jesse Gross <jesse@nicira.com> Fixes: 6728d57 ("dpif-netdev: Translate Geneve options per-flow, not per-packet.") Reported-at: antrea-io/antrea#897 Signed-off-by: Toms Atteka <cpp.code.lv@gmail.com> Acked-by: Ansis Atteka <aatteka@ovn.org> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
This optimization caused FLOW_TNL_F_UDPIF flag not to be used in hash calculation for geneve tunnel when revalidating flows which resulted in different cache hash values and incorrect behaviour. Added test to prevent regression. CC: Jesse Gross <jesse@nicira.com> Fixes: 6728d57 ("dpif-netdev: Translate Geneve options per-flow, not per-packet.") Reported-at: antrea-io/antrea#897 Signed-off-by: Toms Atteka <cpp.code.lv@gmail.com> Acked-by: Ansis Atteka <aatteka@ovn.org> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
This optimization caused FLOW_TNL_F_UDPIF flag not to be used in hash calculation for geneve tunnel when revalidating flows which resulted in different cache hash values and incorrect behaviour. Added test to prevent regression. CC: Jesse Gross <jesse@nicira.com> Fixes: 6728d57 ("dpif-netdev: Translate Geneve options per-flow, not per-packet.") Reported-at: antrea-io/antrea#897 Signed-off-by: Toms Atteka <cpp.code.lv@gmail.com> Acked-by: Ansis Atteka <aatteka@ovn.org> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
One interesting fix is to Geneve tunnel support in the userspace datapath. See antrea-io#897. Signed-off-by: Antonin Bas <abas@vmware.com>
One interesting fix is to Geneve tunnel support in the userspace datapath. See #897. Signed-off-by: Antonin Bas <abas@vmware.com>
A patch was merged in upstream OVS and we picked up this patch when we upgraded to OVS 2.15.1. Fixes antrea-io#897 Signed-off-by: Antonin Bas <abas@vmware.com>
A patch was merged in upstream OVS and we picked up this patch when we upgraded to OVS 2.15.1. Fixes #897 Signed-off-by: Antonin Bas <abas@vmware.com>
Describe the bug
Netpol test suite which tests network policy feature fails if run in Kind cluster with tunnel type set to Geneve. When tunnel type is set to VXLAN, the same set of tests pass.
To Reproduce
Run Netpol suite in Kind cluster created with tunnel type set to Geneve.
Expected
Tests should pass and not depend on tunnel type.
The text was updated successfully, but these errors were encountered: