ISSUE-16469: Duplication of data occurs when send from device to pulsar with messageid different but timestamp exactly same upto millisec

[sijie]

Original Issue: apache#16469

---

Expected behavior

The number of message received in pulsar should be same as the what the total number of message device send.

Actual behavior

1. We send the message from our device to pulsar via websocket.

2. On our device log, we had already send out 2 message within time 2022-07-08 09:14:12 and the status response showing ok means the message already successfully send to the pulsar as shown in below screenshot.
   

3. But when we check into the pulsar topic the duplication of message occurs as shown in below screenshot.
   

4. We found that the message that duplicate have different messageID but same publish timestamp upto millisec. It is impossible that our device sent 2 messages within 1 milisec so we suspect that the pulsar have occurs one duplication of data.

5. This issue occurs 1/5000 message that we send throughout the test. Do you have any ideas about this issues?

System configuration

Pulsar version: 2.7.0
OS: Window

Additional Information

• suspected related to the old issues : Entry duplicated in pulsar when bookie failed and zookeeper disconnected apache/pulsar#7490