New testing framework

[Naman-Priyadarshi]

Currently, BenchExec's tests are executed using nose, which has been inactive for a long time and has a new successor called nose2.

I have compared py.test and nose2 and have found some major differences which make me prefer py.test over nose2. For context read this reddit thread

I have written and implemented tests using py.test for an academic open source project and it was easy to learn and understand. Most of the open source scientific computational tools like pvlib and vector I have contributed for before uses py.test as they can be easy to understand and can have amazing features!

Also we can use codecov which can help us find untested code and make it easy to find issues!

Would love to get your views on this!

[samadpls]

I'm interested in contributing to help address these problems. I have experience with Python and I'm excited to use my skills to improve the testing framework of the project.

I understand that the current testing framework uses nose, which is no longer maintained. I agree that it would be a good idea to evaluate alternative testing frameworks like nose2 or py.test, and I'd be happy to help with this evaluation process. Additionally, I'd like to contribute to improving the current test infrastructure, such as restructuring tests that are problematic for the new framework and adjusting existing tests where necessary.

Please let me know if there are any specific tasks related to the testing infrastructure that I can work on. I'm excited to be part of the project and contribute to its success.

Also, could you provide me with more details about the project and how I can contact you or the team to get started?

Thank you for considering my interest in contributing.

[Naman-Priyadarshi]

Hi @samadpls ! I'm not a maintainer for this project, I'm also a newbie to the project trying to get selected as a GSoC 2023 contributor! I think you should introduce yourself in the comments on the "Welcome to BenchExec Discussions! #902" Discussion

[samadpls]

Hi @samadpls ! I'm not a maintainer for this project, I'm also a newbie to the project trying to get selected as a GSoC 2023 contributor! I think you should introduce yourself in the comments on the "Welcome to BenchExec Discussions! #902" Discussion

Cool, nice to meet you! I hope you get selected for a GSoC project. Best of luck!

[PhilippWendler]

Please let me know if there are any specific tasks related to the testing infrastructure that I can work on. I'm excited to be part of the project and contribute to its success.

I think the current issues with the tests are listed in #162. Just ask there if you have concrete questions on this.

Also, could you provide me with more details about the project and how I can contact you or the team to get started?

For contacting the team, this Discussions place is the best location.

For more details about the project, besides our Readme we also have this paper and slides. Please let me know if you have any further questions!

[PhilippWendler]

I have compared py.test and nose2 and have found some major differences which make me prefer py.test over nose2. For context read this reddit thread
[...]
Would love to get your views on this!

py.test is certainly a strong contender for the to-be-selected framework, but I wonder what would be the actual differences be?

There might also be other potential options besides py.test and nose2, do you know any? We are not limited to considering only these two frameworks. Some numbers on framework popularity would also be nice for decision.

[Naman-Priyadarshi]

The main advantage of py.test over any other framework is

1. The code in py.test is simple, compact, and efficient.
2. The ability to write tests with asset (which can help in major ways)
3. Markers can be used to easily categorize tests
4. Genius ways of parameterizing tests

python's inbuilt unittest is also a known framework and a competitor for py.test and nose2 but pytest is easier to work with.

For unittest, we will have to import modules, create a class and define the testing functions within that class. But for py.test, we only have to define the testing function.

Here are some numbers I found

also these numbers too

nose2

py.test

[PhilippWendler]

Are test classes still supported in py.test? We use them to group similar tests and share common code.

How would the "genius ways of parameterizing tests" look? This might be quite useful for us.

[Naman-Priyadarshi]

Yes, test classes are still supported in py.test!

About the "genius ways", they were just some above average ways of parameterizing tests. which I haven't seen before (I'm still new to parameterizing tests in py.test)

[DaveOkpare]

Parameterizing with pytest can be done either using the @pytest.mark.parametrize decorator or fixtures functions that can be used across various test cases. Also, pytest offers the "genius way of sharing fixtures" using a conftest.py file located in the test root directory, which allows sharing of fixtures for the entire directory without explicitly importing them.

[PhilippWendler]

Thanks! This sounds useful.

[AbhinavMir]

I'd also probably consider a testing framework that enables fuzziness and order independent testing - hypothesis comes to mind. I'm sure PyTest ships with a few plug-ins for this too. Shud probably also think in that direction. :)

Also, remember to have a look at Tox, I remember it being bought up earlier, is probably relevant here.

[airvzxf]

Quick question: Usually, other GSoC have a size which could be 175 hours or 350 hours. Occasionally, they specify two different goals, one for the 175 hours and the other for 350 hours, which is an extension of the first goal (175 hours). I wonder what size are you applying to this project.

[PhilippWendler]

Right. I am not yet decided on this point, I guess both is feasible. I think one deciding factor is how much restructuring/rewriting effort is required by the chosen framework, and this would be discussed in an ideal proposal. Another factor is how much additional suggestions for improvements of our tests a participant has and would be willing to do (for example, adopting parameterized tests, adding some test generation, fuzzers, etc. or just manually writing more tests for modules where coverage is lacking as a part of the project). So if you want to do the 350h, I am pretty sure you would find enough potential work to fill it. :-)

[Naman-Priyadarshi]

I was hoping to work on the project as a long 350h one, I'll be glad if you could guide me in preparing the timeline for the project and make a proper list of deliverables

[PhilippWendler]

Certainly! I would suggest to start investigating and based on this writing a draft, and I will provide feedback, guidance, and where required potential work ideas to fill gaps.

[DaveOkpare]

@PhilippWendler If I may ask, how many people are required for this project?

[PhilippWendler]

This project is certainly not large enough to split it across several people. Note that in Google Summer of Code, project assignments are done only after the application phase, i.e., a project is not reserved just because someone expressed interest and wrote a draft proposal.

[Naman-Priyadarshi]

I have made my first proposal draft and have enabled the comments feature! I would appreciate all the help and guidance. Thank you!

(Weeks 11 and 12 are currently empty in my timeline, and I would appreciate it if you could help me fill this gap.)

[Naman-Priyadarshi]

@PhilippWendler please review it whenever you get the time

[PhilippWendler]

Sure!

[PhilippWendler]

I have done so now.

[Naman-Priyadarshi]

Thank you! I'll look up the comments and make suitable changes

[Naman-Priyadarshi]

I have made some changes to the proposal, and I'm currently working on installing BenchExec on my new Linux system! I will raise an issue if I face any problems in the process.

[Naman-Priyadarshi]

Minor update: I installed the latest Ubuntu 22.04.2 LTS which came with Python 3.10. I followed all the steps to install BenchExec and its dependencies, and I was able to install everything except LXCFS, libseccomp and nose. Also, I wasn't able to configure Cgroups and faced many problems. After installing the basic stuff, I tried running nose, but it wasn't building properly in Python's 3.10 version, so I went to nose's github repo to find out that they solved this issue by recommending nose2 instead of fixing the problem.

So I got tired with everything and decided to install Ubuntu 20.04 in my system, which has Python 3.8 built-in, which worked completely fine with all the dependencies and nose, and I was able to configure Cgroups to work properly! But I'm still facing issues while building LXCFS and libseccomp. I'll devote another day to them, but if I face any other problems, I'll ask for your help!

[PhilippWendler]

Yes, nose unfortunately has problems, one more reason to move away from it. :-) There is a workaround for these problems, that we use in CI. If one has an appropriately old enough version of setuptools and avoids installing the nose wheel, then it works. You can check our CI Docker files and config to see what exactly needs to be done.

LXCFS and libseccomp are optional dependencies of BenchExec, and for the current project not really of relevance.

[Naman-Priyadarshi]

Sorry for the late update; I got busy with school work. I have installed everything properly, and all the tests are passing successfully. I've begun work on a prototype that will allow us to defy goals, time constraints, and work requirements. I will update you soon and create a pull request as soon as the prototype is ready.

[PhilippWendler]

This is good to hear! I already saw #908 and it works for me as well. How long did it take? I left some more comments on the PR and the Google Doc. If you reply in the latter, please ping me here because Google won't notify me.

[PhilippWendler]

Hi @Naman-Priyadarshi, how is it going? Will you be able to finalize and submit the proposal?

[Naman-Priyadarshi]

Yes! I'll complete my proposal as soon as possible and submit it before the deadline!

[Naman-Priyadarshi]

Hello everyone! I have finalized and submitted my GSoC proposal and you can find it here. Thank you everyone for your guidance and help!

[PhilippWendler]

Very nice! Thanks a lot!

[PhilippWendler]

This is implemented by #1043.