apisix转发流量错误

[candicandi]

Issue description

apisix按照如下配置时，有一定几率会转发流量错误

Environment

• apisix version (2.1):
• OS: centos7.8

描述

三个域名对应三个route，假定a b c,两个上游 ，假定 x y，其中xy都只有一个节点，roundrobin,权重1，对应关系为 a->x ,b c ->y, 在转发流量的过程中，当qps在>50之后，约有8%-15%的流量会转发错误，即a->y,b->x，经验证x，y获取到的所有header是客户正常发送的，apisix却分派错了

具体配置如下

upstreams：

"action":"get",
"node":{
"nodes":[
{
"createdIndex":2,
"modifiedIndex":2,
"key":"\/apisix\/upstreams\/333621554711626091",
"value":{
"nodes":[
{
"host":"172.16.27.9",
"port":8088,
"weight":1
}
],
"id":"333621554711626091",
"name":"ceshi-upstream",
"update_time":1608383542,
"type":"roundrobin",
"create_time":1608383542,
"timeout":{
"connect":6000,
"read":6000,
"send":6000
}
}
},
{
"createdIndex":3,
"modifiedIndex":3,
"key":"\/apisix\/upstreams\/333621685657796971",
"value":{
"nodes":[
{
"host":"172.16.27.2",
"port":8087,
"weight":1
}
],
"id":"333621685657796971",
"name":"dz11ht-upstream",
"update_time":1608383620,
"type":"roundrobin",
"create_time":1608383620,
"timeout":{
"connect":6000,
"read":6000,
"send":6000
}
}
},
{
"createdIndex":4,
"modifiedIndex":4,
"key":"\/apisix\/upstreams\/333621748001931627",
"value":{
"nodes":[
{
"host":"172.16.27.2",
"port":8086,
"weight":1
}
],
"id":"333621748001931627",
"name":"dz11-front-upstream",
"update_time":1608383657,
"type":"roundrobin",
"create_time":1608383657,
"timeout":{
"connect":6000,
"read":6000,
"send":6000
}
}
}
],
"dir":true,
"createdIndex":10,
"key":"\/apisix\/upstreams",
"modifiedIndex":76
},
"count":"4",
"header":{
"raft_term":"2",
"member_id":"10276657743932975437",
"revision":"86",
"cluster_id":"14841639068965178418"
}
}

route

{
"action":"get",
"node":{
"nodes":[
{
"createdIndex":5,
"modifiedIndex":20,
"key":"\/apisix\/routes\/333621797343723883",
"value":{
"id":"333621797343723883",
"vars":{
},
"methods":[
"GET",
"HEAD",
"POST",
"PUT",
"DELETE",
"OPTIONS",
"PATCH"
],
"hosts":[
"ceshi.abc.com"
],
"uris":[
"\/*"
],
"update_time":1608384782,
"create_time":1608383687,
"upstream_id":"333621554711626091",
"name":"ceshi-route"
}
},
{
"createdIndex":6,
"modifiedIndex":6,
"key":"\/apisix\/routes\/333622024255570283",
"value":{
"id":"333622024255570283",
"vars":{
},
"methods":[
"GET",
"HEAD",
"POST",
"PUT",
"DELETE",
"OPTIONS",
"PATCH"
],
"hosts":[
"dz11ht.abc.com"
],
"uris":[
"\/*"
],
"update_time":1608383822,
"create_time":1608383822,
"upstream_id":"333621685657796971",
"name":"dz1-route"
}
},
{
"createdIndex":7,
"modifiedIndex":7,
"key":"\/apisix\/routes\/333622061215777131",
"value":{
"id":"333622061215777131",
"vars":{
},
"methods":[
"GET",
"HEAD",
"POST",
"PUT",
"DELETE",
"OPTIONS",
"PATCH"
],
"hosts":[
"dzhtabc.com"
],
"uris":[
"\/*"
],
"update_time":1608383844,
"create_time":1608383844,
"upstream_id":"333621685657796971",
"name":"httest-api-route"
}
},
{
"createdIndex":21,
"modifiedIndex":21,
"key":"\/apisix\/routes\/333623666443026795",
"value":{
"id":"333623666443026795",
"vars":{
},
"methods":[
"GET",
"HEAD",
"POST",
"PUT",
"DELETE",
"OPTIONS",
"PATCH"
],
"hosts":[
"test.abc.com"
],
"uris":[
"\/*"
],
"update_time":1608384801,
"create_time":1608384801,
"upstream_id":"333621554711626091",
"name":"test-api-route"
}
},
{
"createdIndex":34,
"modifiedIndex":35,
"key":"\/apisix\/routes\/333627619691987307",
"value":{
"id":"333627619691987307",
"vars":{
},
"methods":[
"GET",
"HEAD",
"POST",
"PUT",
"DELETE",
"OPTIONS",
"PATCH"
],
"hosts":[
"dz11.abccom"
],
"uris":[
"\/*"
],
"update_time":1608388794,
"create_time":1608387157,
"upstream_id":"333621748001931627",
"name":"dzfw-route"
}
}
],
"dir":true,
"createdIndex":9,
"key":"\/apisix\/routes",
"modifiedIndex":75
},
"count":"6",
"header":{
"raft_term":"2",
"member_id":"10276657743932975437",
"revision":"86",
"cluster_id":"14841639068965178418"
}
}

What's the actual result? (including assertion message & call stack if applicable)

What's the expected result?

[candicandi]

配置文件是按照github上的配置文件，未做什么改动

[moonming]

@membphis @nic-chen @spacewander

[candicandi]

可以联系我1083796027提供两台线上的机子，你们可以查看调试日志等

[nic-chen]

可以联系我1083796027提供两台线上的机子，你们可以查看调试日志等

OK

[membphis]

@nic-chen I think we have caught the bug, you can leave a message here.

[curvesoft]

我也遇到了同样的问题，目前解决了么？

[membphis]

我也遇到了同样的问题，目前解决了么？

you can make a try with this PR: #3105