Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QoS] Support dynamic headroom calculation for Barefoot platforms #6151

Open
wants to merge 20 commits into
base: master
Choose a base branch
from

Conversation

MariuszStachura
Copy link

Signed-off-by: Mariusz Stachura mariusz.stachura@intel.com

What I did
Adding the dynamic headroom calculation support for Barefoot platforms.

Why I did it
Enabling dynamic mode for barefoot case.

How I verified it
The community tests are adjusted and pass.

Details if related

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Aug 11, 2022

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: MariuszStachura / name: Mariusz Stachura (8381a27)

Signed-off-by: Mariusz Stachura <mariusz.stachura@intel.com>
@lgtm-com
Copy link

lgtm-com bot commented Aug 11, 2022

This pull request introduces 2 alerts when merging 8381a27 into 2e40f50 - view on LGTM.com

new alerts:

  • 2 for Unused local variable

@jarias-lfx
Copy link

/easycla

Do not use PG 5, but stick with 6 as it was enabled. Also
fix check_pool_size for barefoot case.
@MariuszStachura MariuszStachura requested review from stephenxs and removed request for neethajohn August 31, 2022 19:59
@lgtm-com
Copy link

lgtm-com bot commented Aug 31, 2022

This pull request introduces 2 alerts when merging 6a37963 into 00b9f3e - view on LGTM.com

new alerts:

  • 2 for Unused local variable

Copy link
Contributor

@stephenxs stephenxs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comment

tests/qos/test_buffer.py Outdated Show resolved Hide resolved
@@ -1919,10 +1958,13 @@ def _get_max_speed_from_list(speed_list_str):
original_speed = duthost.shell('redis-cli -n 4 hget "PORT|{}" speed'.format(port_to_test))['stdout']
original_cable_length = duthost.shell('redis-cli -n 4 hget "CABLE_LENGTH|AZURE" {}'.format(port_to_test))['stdout']
original_pool_size = duthost.shell('redis-cli hget BUFFER_POOL_TABLE:ingress_lossless_pool size')['stdout']
if DEFAULT_OVER_SUBSCRIBE_RATIO:
if duthost.facts['asic_type'] == 'barefoot':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can introduce a method that returns whether the shared headroom pool is enabled. By doing so the logic can be simplified.

def is_shared_headroom_pool_enabled(duthost):
    return DEFAULT_OVER_SUBSCRIBE_RATIO or duthost.facts['asic_type'] in ['barefoot']

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this condition as it is not needed anymore. Thanks!

MariuszStachura and others added 2 commits September 2, 2022 11:29
Co-authored-by: Stephen Sun <5379172+stephenxs@users.noreply.github.com>
@yejianquan
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

stephenxs
stephenxs previously approved these changes Sep 2, 2022
Copy link
Contributor

@stephenxs stephenxs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Author

@MariuszStachura MariuszStachura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EDIT: Added by mistake

Copy link
Author

@MariuszStachura MariuszStachura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not disable loganalyzer

It was failing for me in this test, but probably it should not be disabled.

@MariuszStachura
Copy link
Author

Sorry @stephenxs, I did not dismissed your review, I was just testing github.dev and edit this file with that and this caused this automatic dismissal.
The same issue as in isaacs/github#1157

@yejianquan
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@stephenxs
Copy link
Contributor

stephenxs commented Sep 2, 2022

Sorry @stephenxs, I did not dismissed your review, I was just testing github.dev and edit this file with that and this caused this automatic dismissal. The same issue as in isaacs/github#1157

no problem. this is the way it works. just let me know once you finish all the changes.

@neethajohn
Copy link
Contributor

@MariuszStachura, there are a lot of changes in this PR touching multiple things. please split this into 2 PRs - one for qos sai test alone and other for dynamic headroom calculation

@MariuszStachura
Copy link
Author

MariuszStachura commented Sep 23, 2022

@neethajohn

there are a lot of changes in this PR touching multiple things. please split this into 2 PRs - one for qos sai test alone and other for dynamic headroom calculation

Ok, this one is dynamic headroom calculation only, pasting the result:
_- test_change_speed_cable[50000-1500-15m] PASS

  • test_change_speed_cable[50000-1500-40m] PASS
  • test_change_speed_cable[50000-9100-15m] PASS
  • test_change_speed_cable[50000-9100-40m] PASS
  • test_change_speed_cable[10000-1500-15m] PASS
  • test_change_speed_cable[10000-1500-40m] PASS
  • test_change_speed_cable[10000-9100-15m] PASS
  • test_change_speed_cable[10000-9100-40m] PASS
  • test_headroom_override PASS
  • test_shared_headroom_pool_configure SKIPPED (intentionally)- over_subscribe_ratio is not supported for Barefoot platforms yet. We configure shared headroom/buffer model by default (no possibility to toggle it). It is enabled by size.
  • test_lossless_pg[3-4] PASS
  • test_lossless_pg[6] PASS
  • test_port_admin_down SKIPPED (intentionally)- we do not support enhanced dynamic buffer calculation (administratively down ports) yet. Test is skipped due to the lack of extra overhead param defined.
  • test_port_auto_neg PASS
  • test_exceeding_headroom SKIPPED (intentionally)- Barefoot platforms do not have a limitation for maximum port headroom. Test is skipped just after the cable length has been increased to 2 km.
  • test_buffer_model_test PASS
  • test_buffer_deployment PASS_

This PR relates on:
sonic-net/sonic-buildimage#11708
sonic-net/sonic-utilities#2306
sonic-net/sonic-swss#2412

Thank you!

@StormLiangMS
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@azure-pipelines
Copy link

The pre-commit check detected issues in the files touched by this pull request.
The detected issues may be old or new. For new issues, please try to fix them.

For old issues, it is not mandatory to fix them because they were not caused by this change. It is unfair to blame
author of this pull request. But if you can take extra effort to fix the old issues as well, that would be great!

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/qos/test_buffer.py:2:1: F401 'os' imported but unused
tests/qos/test_buffer.py:3:1: F401 'sys' imported but unused
tests/qos/test_buffer.py:14:1: F401 'tests.common.broadcom_data.is_broadcom_device' imported but unused
tests/qos/test_buffer.py:17:1: F401 'tests.common.fixtures.conn_graph_facts.conn_graph_facts' imported but unused
tests/qos/test_buffer.py:23:1: F401 'tests.common.dualtor.dual_tor_utils.dualtor_ports' imported but unused
tests/qos/test_buffer.py:23:91: E261 at least two spaces before inline comment
tests/qos/test_buffer.py:64:1: E302 expected 2 blank lines, found 1
tests/qos/test_buffer.py:65:121: E501 line too long (137 > 120 characters)
tests/qos/test_buffer.py:76:121: E501 line too long (122 > 120 characters)
tests/qos/test_buffer.py:96:121: E501 line too long (136 > 120 characters)
tests/qos/test_buffer.py:139:1: E302 expected 2 blank lines, found 1
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@azure-pipelines
Copy link

The pre-commit check detected issues in the files touched by this pull request.
The detected issues may be old or new. For new issues, please try to fix them.

For old issues, it is not mandatory to fix them because they were not caused by this change. It is unfair to blame
author of this pull request. But if you can take extra effort to fix the old issues as well, that would be great!

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/qos/test_buffer.py:12:1: F401 'tests.common.broadcom_data.is_broadcom_device' imported but unused
tests/qos/test_buffer.py:143:1: E302 expected 2 blank lines, found 1
tests/qos/test_buffer.py:451:121: E501 line too long (127 > 120 characters)
tests/qos/test_buffer.py:491:121: E501 line too long (125 > 120 characters)
tests/qos/test_buffer.py:524:121: E501 line too long (178 > 120 characters)
tests/qos/test_buffer.py:531:121: E501 line too long (140 > 120 characters)
tests/qos/test_buffer.py:538:121: E501 line too long (121 > 120 characters)
tests/qos/test_buffer.py:546:121: E501 line too long (176 > 120 characters)
tests/qos/test_buffer.py:551:121: E501 line too long (134 > 120 characters)
tests/qos/test_buffer.py:557:121: E501 line too long (126 > 120 characters)
tests/qos/test_buffer.py:560:19: E225 missing whitespace around operator
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

from tests.common.mellanox_data import is_mellanox_device
from tests.common.innovium_data import is_innovium_device
from tests.common.plugins.loganalyzer.loganalyzer import LogAnalyzer
from tests.common.utilities import check_qos_db_fv_reference_with_table
from tests.common.utilities import skip_release
from tests.common.dualtor.dual_tor_utils import is_tunnel_qos_remap_enabled, dualtor_ports # lgtm[py/unused-import]
from tests.common.dualtor.dual_tor_utils import is_tunnel_qos_remap_enabled
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add back the function 'dualtor_ports'

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, this was to fix:
tests/qos/test_buffer.py:23:1: F401 'tests.common.dualtor.dual_tor_utils.dualtor_ports' imported but unused

dualtor_ports is used inside:
def test_buffer_deployment(duthosts, rand_one_dut_hostname, conn_graph_facts, tbinfo, dualtor_ports):
and there it is passed as an argument.

@@ -918,7 +929,7 @@ def test_change_speed_cable(duthosts, rand_one_dut_hostname, conn_graph_facts, p
if speed_to_test == original_speed and cable_len_to_test == original_cable_len:
pytest.skip('Speed, MTU and cable length matches the default value, nothing to test, skip')
expected_profile = make_expected_profile_name(speed_to_test, cable_len_to_test)
if duthost.shell('redis-cli hget BUFFER_PROFILE_TABLE:{}'.format(expected_profile))['stdout']:
if duthost.shell('redis-cli keys BUFFER_PROFILE_TABLE:{}'.format(expected_profile))['stdout'] != '':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this changed to 'keys'?

Copy link
Author

@MariuszStachura MariuszStachura Oct 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was failing for me and I thought it must have been a mistake:

admin@sonic:~$ redis-cli keys BUFFER_PROFILE_TABLE:expected_profile
(empty array)
admin@sonic:~$ redis-cli hget BUFFER_PROFILE_TABLE:expected_profile
(error) ERR wrong number of arguments for 'hget' command

hget always failed

Comment on lines 2436 to 2437
buffer_table_up[KEY_2_LOSSLESS_QUEUE][1] = ('BUFFER_QUEUE_TABLE', '0-2', '[BUFFER_PROFILE_TABLE:]')
buffer_table_up[KEY_2_LOSSLESS_QUEUE][3] = ('BUFFER_QUEUE_TABLE', '5-6', '[BUFFER_PROFILE_TABLE:]')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The profile name is missing here

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this was intentional for BUFFER_QUEUE_TABLE in our case, thank you for thorough review, I'll try to fix pre-commit issues locally first to not trigger jobs again and again.

@azure-pipelines
Copy link

The pre-commit check detected issues in the files touched by this pull request.
The detected issues may be old or new. For new issues, please try to fix them.

For old issues, it is not mandatory to fix them because they were not caused by this change. It is unfair to blame
author of this pull request. But if you can take extra effort to fix the old issues as well, that would be great!

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/qos/test_buffer.py:580:121: E501 line too long (140 > 120 characters)
tests/qos/test_buffer.py:589:121: E501 line too long (121 > 120 characters)
tests/qos/test_buffer.py:599:121: E501 line too long (132 > 120 characters)
tests/qos/test_buffer.py:604:121: E501 line too long (134 > 120 characters)
tests/qos/test_buffer.py:610:121: E501 line too long (126 > 120 characters)
tests/qos/test_buffer.py:638:121: E501 line too long (134 > 120 characters)
tests/qos/test_buffer.py:642:121: E501 line too long (122 > 120 characters)
tests/qos/test_buffer.py:653:58: E712 comparison to True should be 'if cond is True:' or 'if cond:'
tests/qos/test_buffer.py:737:121: E501 line too long (122 > 120 characters)
tests/qos/test_buffer.py:755:58: E712 comparison to True should be 'if cond is True:' or 'if cond:'
tests/qos/test_buffer.py:763:121: E501 line too long (130 > 120 characters)
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@lgtm-com
Copy link

lgtm-com bot commented Oct 19, 2022

This pull request fixes 4 alerts when merging b3a6b2a into 79ffe4c - view on LGTM.com

fixed alerts:

  • 4 for Unused import

@azure-pipelines
Copy link

The pre-commit check detected issues in the files touched by this pull request.
The detected issues may be old or new. For new issues, please try to fix them.

For old issues, it is not mandatory to fix them because they were not caused by this change. It is unfair to blame
author of this pull request. But if you can take extra effort to fix the old issues as well, that would be great!

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/qos/test_buffer.py:580:121: E501 line too long (140 > 120 characters)
tests/qos/test_buffer.py:589:121: E501 line too long (121 > 120 characters)
tests/qos/test_buffer.py:599:121: E501 line too long (132 > 120 characters)
tests/qos/test_buffer.py:744:121: E501 line too long (122 > 120 characters)
tests/qos/test_buffer.py:782:121: E501 line too long (176 > 120 characters)
tests/qos/test_buffer.py:785:121: E501 line too long (176 > 120 characters)
tests/qos/test_buffer.py:804:121: E501 line too long (130 > 120 characters)
tests/qos/test_buffer.py:963:121: E501 line too long (140 > 120 characters)
tests/qos/test_buffer.py:1074:121: E501 line too long (164 > 120 characters)
tests/qos/test_buffer.py:1114:121: E501 line too long (153 > 120 characters)
tests/qos/test_buffer.py:1153:121: E501 line too long (122 > 120 characters)
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@MariuszStachura
Copy link
Author

@neethajohn I did what I could regarding the pre-commit issues. Only issues left are about lines being too long, but I couldn't fix them in a nice way.

@azure-pipelines
Copy link

The pre-commit check detected issues in the files touched by this pull request.
The detected issues may be old or new. For new issues, please try to fix them.

For old issues, it is not mandatory to fix them because they were not caused by this change. It is unfair to blame
author of this pull request. But if you can take extra effort to fix the old issues as well, that would be great!

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/qos/test_buffer.py:580:121: E501 line too long (140 > 120 characters)
tests/qos/test_buffer.py:589:121: E501 line too long (121 > 120 characters)
tests/qos/test_buffer.py:599:121: E501 line too long (132 > 120 characters)
tests/qos/test_buffer.py:744:121: E501 line too long (122 > 120 characters)
tests/qos/test_buffer.py:782:121: E501 line too long (176 > 120 characters)
tests/qos/test_buffer.py:785:121: E501 line too long (176 > 120 characters)
tests/qos/test_buffer.py:804:121: E501 line too long (130 > 120 characters)
tests/qos/test_buffer.py:963:121: E501 line too long (140 > 120 characters)
tests/qos/test_buffer.py:1074:121: E501 line too long (164 > 120 characters)
tests/qos/test_buffer.py:1114:121: E501 line too long (153 > 120 characters)
tests/qos/test_buffer.py:1153:121: E501 line too long (122 > 120 characters)
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@lgtm-com
Copy link

lgtm-com bot commented Oct 19, 2022

This pull request fixes 9 alerts when merging fb31e77 into 79ffe4c - view on LGTM.com

fixed alerts:

  • 4 for Unused local variable
  • 4 for Unused import
  • 1 for Testing equality to None

@MariuszStachura
Copy link
Author

Hi @StormLiangMS here are the actual changes:
https://github.com/sonic-net/sonic-mgmt/pull/6151/files/78ed3dc117a659a67ee574d75e506dd53aa4c9e9
Without pre-format fixes that are mostly line wrapping and unused variable fixes.

@MariuszStachura
Copy link
Author

Hello @XuChen-MSFT, could we get these tests merged? Many thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants