Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pytest] improvements for pytest infrastructure #2

Closed
wants to merge 24 commits into from
Closed

[pytest] improvements for pytest infrastructure #2

wants to merge 24 commits into from

Conversation

stepanblyschak
Copy link
Owner

@stepanblyschak stepanblyschak commented Jul 25, 2019 •
edited
Loading

Description of PR

Summary:

  • pytest:
    • W/A some issues with pytest-ansible plugin and provided fixtures
    • add ptf adapter package and fixture to send/receive PTF traffic controlled from pytest
    • translate fdb test to use PTF adapter
    • Divide LLDP test into two test cases
  • ansible
    • add pytest_runner.yml to wrap pytest in ansible run
    • remove lldp.yml test code ansible and lldp_neighbor.yml. Instead lldp.yml will run pytest test_lldp.py

Fixes # (issue)

Type of change

  • [] Bug fix
  • Testbed and Framework(new/improvement)
  • [] Test case(new/improvement)

Approach

How did you do it?

How did you verify/test it?

stepanb@62c67c04c2bd:/stepanb/sonic-mgmt/tests$ py.test -k 'not platform'  --testbed=$SWITCH-t0 --testbed_file=testbed.csv --inventory=../ansible/inventory --host-pattern $SWITCH-t0 -v --ignore=ptftests
=========================================================================================================== test session starts ===========================================================================================================
platform linux2 -- Python 2.7.12, pytest-4.6.4, py-1.8.0, pluggy-0.12.0 -- /usr/bin/python
cachedir: .pytest_cache
Using --randomly-seed=1564067815
ansible: 2.0.0.2
rootdir: /stepanb/sonic-mgmt/tests, inifile: pytest.ini
plugins: randomly-1.2.3, ansible-2.0.2
collected 16 items / 11 deselected / 5 selected                                                                                                                                                                                           

test_bgp_fact.py::test_bgp_facts PASSED                                                                                                                                                                                             [ 20%]
test_lldp.py::test_lldp PASSED                                                                                                                                                                                                      [ 40%]
test_lldp.py::test_lldp_neighbor PASSED                                                                                                                                                                                             [ 60%]
test_bgp_speaker.py::test_bgp_speaker SKIPPED                                                                                                                                                                                       [ 80%]
fdb/test_fdb.py::test_fdb PASSED                                                                                                                                                                                                    [100%]

========================================================================================== 4 passed, 1 skipped, 11 deselected in 147.06 seconds ===========================================================================================

Verify ansible LLDP test

ansible-playbook test_sonic.yml -i inventory -l r-boxer-sw01 -e testbed_name=r-boxer-sw01-t0 -e topo=t0 -e testcase_name=lldp -vvvvvv

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Stepan Blyschak added 20 commits July 25, 2019 13:40
[tests/conftest.py] override pytest-ansible fixtures to overcome scop…
b53b1cc 
…e issues

There are two issues with pytest-ansible I found:
- ansible_adhoc is limited to function scope which makes it not usable for wider scope test fixtures
- localhost fixture has some hidden dependency on ansible_adhoc (even without changing default scope)
  e.g.:

   def test_x(ansible_adhoc, testbed):
       duthost = AnsibleHost(ansible_adhoc, testbed['dut'])
       pass

   def test_y(ansible_adhoc, localhost, testbed):
       pass

  This snippet of test code fails to evaluate localhost fixture in test_y:

  test_x.py::test_x PASSED                                                                                                                                                                                                            [ 50%]
  test_x.py::test_y ERROR                                                                                                                                                                                                             [100%]

  Part of error log:

        host = self.get_host(hostname)
        if host is None:
>           raise AnsibleError("no host vars as host is not in inventory: %s" % hostname)
E           AnsibleError: ERROR! no host vars as host is not in inventory: arc-switch1025

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[pytest-infra] introduce PtfTestAdapter and ptfadapter fixture to con…
329e67c 
…trol PTF traffic from sonic-mgmt node

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[ptfadapter] add default ptf config
a003a28 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[conftest] move overridden ansible fixture to seperate file
d5830e1 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[ptf_runner] rename ptf.py to ptf_runner.py to avoid conflict with pt…
7221045 
…f framework

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[gitignore] update with cpython compiled files
9328407 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/fdb] rewrite fdb test with ptfadapter & fix change_mac.sh script
e890945 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/ansible_host] rename to AnsibleHost to meet python naming conv…
a930acb 
…ention

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/ptfadapter] move qlen/relax to PtfTestAdapter object
1f335c6 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/ptfadapter] non need for disable_* config keys for newer ptf
6be1b29 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests] change ansible_host to AnsibleHost
45aa8b9 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[conftest] remove extra new line
37054a0 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/fdb] reduce info to debug
f80916f 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/fdb] cleanup FDB test
bcf541d 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/ptfadapter] correct on the comment
6a1cba1 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[pytest] introduce pytest_runner playbook
e4a7bfe 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[test_lldp] divide into two test cases and put a TODO about SNMP comm…
bebc1f4 
…unity string

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[lldp] call py.test to run lldp test; from now on any change/new test…
7911f51 
… cases for lldp should be done in pytest

Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/bgp_speaker] skip until test is fixed
46eece9 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/ptfadapter] add README.md on how to use ptfadapter
d6be436 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
@wangxin
Copy link

wangxin commented Jul 26, 2019

Regarding tests/fdb/init.py, is it necessary to make the fdb testing scripts a package?

wangxin
wangxin reviewed Jul 26, 2019
View reviewed changes
tests/fdb/test_fdb.py Outdated
res = duthost.command('show mac')
logger.debug('"show mac" output on DUT')
for line in res['stdout_lines']:
logger.debug(' {}'.format(line))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Package pprint is for this kind of job.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I replaced with pprint

Stepan Blyschak added 4 commits July 29, 2019 15:18
[.gitignore] ignore pytest_cached
bfb27c3 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/conftest] add shortcut fixtures for DUT and PTF
a6847d3 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/fdb] remove __init__.py from fdb test
20150f1 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
[tests/fdb] use pprint
0b5c57d 
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
@stepanblyschak
Copy link
Owner Author

Regarding tests/fdb/init.py, is it necessary to make the fdb testing scripts a package?

No, thanks

stepanblyschak pushed a commit that referenced this pull request Nov 4, 2019
anaconda testbed. 40G #2
2fc814c 
Signed-off-by: Roman Kachur <romankac@mellanox.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wangxin wangxin wangxin left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@stepanblyschak @wangxin