Unit tests don't seem to work

[IanBoden]

Prerequisites

Note: You may remove this section prior to submitting your report.

A small team of volunteers monitors issues. Please help us to help you by making it simple to understand and, if possible,
replicate your issue. Prior to reporting a bug please:

• Test the latest release of the library.
• Search existing issues.
• Read the relevant documentation.
• Review your server configuration and logs.
• Consider testing against a different server (e.g. mqtt.eclipseprojects.io or test.mosquitto.org)
• If possible, test using another tool (e.g. MQTTX / mosquitto_sub)
  to confirm the issue is specific to this client.
• If you are unsure if you have found a bug, please consider asking on stackoverflow for a quicker response.

Bug Description

Please provide a clear and concise description of the bug.

The contributing simply says:

Ensure that all new and existing tests pass by running tox

But as far as I can see this is impossible without significant code changes, firstly there is no setup.py which is a requirement of tox. Setting:
skipsdist = True
in the tox.ini at least gets tox to run. So far I've had no luck in any of the tests working though.

The Makefile doesn't use tox at all and running:
make test
does run some tests although with a lot of failures. Basically getting lots of socket timeouts so something isn't happy creating the test server I think.

Reproduction

Please provide detailed steps showing how to replicate the issue (it's difficult to fix an issue we cannot replicate).
If errors are output then include the full error (including any stack trace).
Most issues should include a minimal example that
demonstrates the issue (ideally one that can be run without modification, i.e. runnable code using a public broker).

checkout the code (or make a fork if you prefer)
run tox
get error:

ERROR: No setup.py file found. The expected location is:
/paho/paho.mqtt.python/setup.py
You can

1. Create one:
   https://packaging.python.org/tutorials/distributing-packages/#setup-py
2. Configure tox to avoid running sdist:
   https://tox.readthedocs.io/en/latest/example/general.html#avoiding-expensive-sdist

run make test
get results:

30 failed, 71 passed, 21 skipped, 4 xfailed, 30 errors in 338.79s (0:05:38)

Environment

• Python version: 3.9
• Library version: master
• Operating system (including version): AlmaLinux release 8.9 (Midnight Oncilla)
• MQTT server (name, version, configuration, hosting details):

Logs

For many issues, especially when you cannot provide code to replicate the issue, it's helpful to include logs. Please
consider including:

• library logs; see the readme and logger example.
• broker logs (availability will depend the server in use)

[JamesParrott]

The tox tests ran last week in CI for number #453 on the callbackapi-default branch, and passed.

https://github.com/eclipse/paho.mqtt.python/actions/runs/8590915867

I'm not sure what happened with number #454 on master, but it still passed on 3 python versions.

[IanBoden]

The tox tests ran last week in CI for number #453 on the callbackapi-default branch, and passed.

https://github.com/eclipse/paho.mqtt.python/actions/runs/8590915867

I'm not sure what happened with number #454 on master, but it still passed on 3 python versions.

I had an older version to tox which was causing the problem with tox requiring a setup.py

Upgrading tox allowed it to run but it was still failing a lot of tests.

Moving to almalinux9 and running without paho.mqtt.testing I get the same results as the run you posted just without the v5 tests, Adding paho.mqtt.testing I'm getting occasional failures or hangs in v5, but about 50% of the time it runs clean.

[PierreF]

I think this issue is a duplicate of #789. Tests are known to fail from time to time on mqtt v5 test (which require paho.mqtt.testing).
I think I've fixed them in 2.1.0 (which will be released in the next hours).