[Arista] Disable ATA NCQ for a few products #13739

Staphylo · 2023-02-09T15:53:37Z

Why I did it

Some products might experience an occasional IO failure in the communication between CPU and SSD.
Based on some research it could be attributable to some device not handling ATA NCQ (Native Command Queue).

This issue currently affect 4 products:

DCS-7170-32C*
DCS-7170-64C
DCS-7060DX4-32
DCS-7260CX3-64

How I did it

This change disable NCQ on the affected drive for a small set of products.

How to verify it

When the fix is applied, these 2 patterns can be found in the dmesg.
ata1.00: FORCE: horkage modified (noncq)
NCQ (not used)

Test results using: fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4

with NCQ (ata1.00: 61865984 sectors, multi 1: LBA48 NCQ (depth 32), AA)

   READ: bw=33.9MiB/s (35.6MB/s), 33.9MiB/s-33.9MiB/s (35.6MB/s-35.6MB/s), io=4073MiB (4270MB), run=120078-120078msec
  WRITE: bw=34.1MiB/s (35.8MB/s), 34.1MiB/s-34.1MiB/s (35.8MB/s-35.8MB/s), io=4100MiB (4300MB), run=120078-120078msec

without NCQ (ata1.00: 61865984 sectors, multi 1: LBA48 NCQ (not used))

   READ: bw=31.7MiB/s (33.3MB/s), 31.7MiB/s-31.7MiB/s (33.3MB/s-33.3MB/s), io=3808MiB (3993MB), run=120083-120083msec
  WRITE: bw=31.9MiB/s (33.4MB/s), 31.9MiB/s-31.9MiB/s (33.4MB/s-33.4MB/s), io=3830MiB (4016MB), run=120083-120083msec

Which release branch to backport (provide reason below if selected)

Description for the changelog

Disable ATA NCQ for a few Arista products

Staphylo · 2023-02-09T15:55:06Z

@prgeor

files/Aboot/boot0.j2

prgeor · 2023-02-10T05:08:23Z

@Staphylo can you update the test/performance result

Some products might experience an occasional IO failure in the communication between CPU and SSD. Based on some research it could be attributable to some device not handling ATA NCQ (Native Command Queue). This change disable NCQ on the affected drive for a set of products. When the fix is applied, these 2 patterns can be found in the dmesg. ata1.00: FORCE: horkage modified (noncq) NCQ (not used)

Staphylo · 2023-02-10T14:57:25Z

@Staphylo can you update the test/performance result

I'll add these to the description

prgeor · 2023-02-15T18:31:22Z

@lguohan please help merge

prgeor · 2023-02-15T22:10:52Z

@StormLiangMS please merge 202211

prgeor · 2023-02-15T22:11:05Z

@yxieca please merge 202205

prgeor · 2023-02-15T22:13:19Z

@qiluo-msft please merge to 202012

Why I did it Some products might experience an occasional IO failure in the communication between CPU and SSD. Based on some research it could be attributable to some device not handling ATA NCQ (Native Command Queue). This issue currently affect 4 products: DCS-7170-32C* DCS-7170-64C DCS-7060DX4-32 DCS-7260CX3-64 How I did it This change disable NCQ on the affected drive for a small set of products. How to verify it When the fix is applied, these 2 patterns can be found in the dmesg. ata1.00: FORCE: horkage modified (noncq) NCQ (not used) Test results using: fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 with NCQ (ata1.00: 61865984 sectors, multi 1: LBA48 NCQ (depth 32), AA) READ: bw=33.9MiB/s (35.6MB/s), 33.9MiB/s-33.9MiB/s (35.6MB/s-35.6MB/s), io=4073MiB (4270MB), run=120078-120078msec WRITE: bw=34.1MiB/s (35.8MB/s), 34.1MiB/s-34.1MiB/s (35.8MB/s-35.8MB/s), io=4100MiB (4300MB), run=120078-120078msec without NCQ (ata1.00: 61865984 sectors, multi 1: LBA48 NCQ (not used)) READ: bw=31.7MiB/s (33.3MB/s), 31.7MiB/s-31.7MiB/s (33.3MB/s-33.3MB/s), io=3808MiB (3993MB), run=120083-120083msec WRITE: bw=31.9MiB/s (33.4MB/s), 31.9MiB/s-31.9MiB/s (33.4MB/s-33.4MB/s), io=3830MiB (4016MB), run=120083-120083msec Which release branch to backport (provide reason below if selected)

mssonicbld · 2023-02-17T03:11:39Z

Cherry-pick PR to 202205: #13842

Why I did it Some products might experience an occasional IO failure in the communication between CPU and SSD. Based on some research it could be attributable to some device not handling ATA NCQ (Native Command Queue). This issue currently affect 4 products: DCS-7170-32C* DCS-7170-64C DCS-7060DX4-32 DCS-7260CX3-64 How I did it This change disable NCQ on the affected drive for a small set of products. How to verify it When the fix is applied, these 2 patterns can be found in the dmesg. ata1.00: FORCE: horkage modified (noncq) NCQ (not used) Test results using: fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 with NCQ (ata1.00: 61865984 sectors, multi 1: LBA48 NCQ (depth 32), AA) READ: bw=33.9MiB/s (35.6MB/s), 33.9MiB/s-33.9MiB/s (35.6MB/s-35.6MB/s), io=4073MiB (4270MB), run=120078-120078msec WRITE: bw=34.1MiB/s (35.8MB/s), 34.1MiB/s-34.1MiB/s (35.8MB/s-35.8MB/s), io=4100MiB (4300MB), run=120078-120078msec without NCQ (ata1.00: 61865984 sectors, multi 1: LBA48 NCQ (not used)) READ: bw=31.7MiB/s (33.3MB/s), 31.7MiB/s-31.7MiB/s (33.3MB/s-33.3MB/s), io=3808MiB (3993MB), run=120083-120083msec WRITE: bw=31.9MiB/s (33.4MB/s), 31.9MiB/s-31.9MiB/s (33.4MB/s-33.4MB/s), io=3830MiB (4016MB), run=120083-120083msec Which release branch to backport (provide reason below if selected)

mssonicbld · 2023-02-17T14:20:30Z

Cherry-pick PR to 202211: #13865

Why I did it Some products might experience an occasional IO failure in the communication between CPU and SSD. Based on some research it could be attributable to some device not handling ATA NCQ (Native Command Queue). This issue currently affect 4 products: DCS-7170-32C* DCS-7170-64C DCS-7060DX4-32 DCS-7260CX3-64 How I did it This change disable NCQ on the affected drive for a small set of products. How to verify it When the fix is applied, these 2 patterns can be found in the dmesg. ata1.00: FORCE: horkage modified (noncq) NCQ (not used) Test results using: fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 with NCQ (ata1.00: 61865984 sectors, multi 1: LBA48 NCQ (depth 32), AA) READ: bw=33.9MiB/s (35.6MB/s), 33.9MiB/s-33.9MiB/s (35.6MB/s-35.6MB/s), io=4073MiB (4270MB), run=120078-120078msec WRITE: bw=34.1MiB/s (35.8MB/s), 34.1MiB/s-34.1MiB/s (35.8MB/s-35.8MB/s), io=4100MiB (4300MB), run=120078-120078msec without NCQ (ata1.00: 61865984 sectors, multi 1: LBA48 NCQ (not used)) READ: bw=31.7MiB/s (33.3MB/s), 31.7MiB/s-31.7MiB/s (33.3MB/s-33.3MB/s), io=3808MiB (3993MB), run=120083-120083msec WRITE: bw=31.9MiB/s (33.4MB/s), 31.9MiB/s-31.9MiB/s (33.4MB/s-33.4MB/s), io=3830MiB (4016MB), run=120083-120083msec Which release branch to backport (provide reason below if selected)

Why I did it Fix similar issue seen on #13739 but only for DCS-7050CX3-32S How I did it Add a kernel parameter to tell libata to disable NCQ How to verify it The message ata2.00: FORCE: horkage modified (noncq) should appear on the dmesg. Test results using: fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 with NCQ READ: bw=26.1MiB/s (27.4MB/s), 26.1MiB/s-26.1MiB/s (27.4MB/s-27.4MB/s), io=3136MiB (3288MB), run=120053-120053msec WRITE: bw=26.3MiB/s (27.6MB/s), 26.3MiB/s-26.3MiB/s (27.6MB/s-27.6MB/s), io=3161MiB (3315MB), run=120053-120053msec without NCQ READ: bw=22.0MiB/s (23.1MB/s), 22.0MiB/s-22.0MiB/s (23.1MB/s-23.1MB/s), io=2647MiB (2775MB), run=120069-120069msec WRITE: bw=22.2MiB/s (23.3MB/s), 22.2MiB/s-22.2MiB/s (23.3MB/s-23.3MB/s), io=2665MiB (2795MB), run=120069-120069msec

Why I did it Fix similar issue seen on sonic-net#13739 but only for DCS-7050CX3-32S How I did it Add a kernel parameter to tell libata to disable NCQ How to verify it The message ata2.00: FORCE: horkage modified (noncq) should appear on the dmesg. Test results using: fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 with NCQ READ: bw=26.1MiB/s (27.4MB/s), 26.1MiB/s-26.1MiB/s (27.4MB/s-27.4MB/s), io=3136MiB (3288MB), run=120053-120053msec WRITE: bw=26.3MiB/s (27.6MB/s), 26.3MiB/s-26.3MiB/s (27.6MB/s-27.6MB/s), io=3161MiB (3315MB), run=120053-120053msec without NCQ READ: bw=22.0MiB/s (23.1MB/s), 22.0MiB/s-22.0MiB/s (23.1MB/s-23.1MB/s), io=2647MiB (2775MB), run=120069-120069msec WRITE: bw=22.2MiB/s (23.3MB/s), 22.2MiB/s-22.2MiB/s (23.3MB/s-23.3MB/s), io=2665MiB (2795MB), run=120069-120069msec

Why I did it Fix similar issue seen on #13739 but only for DCS-7050CX3-32S How I did it Add a kernel parameter to tell libata to disable NCQ How to verify it The message ata2.00: FORCE: horkage modified (noncq) should appear on the dmesg. Test results using: fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 with NCQ READ: bw=26.1MiB/s (27.4MB/s), 26.1MiB/s-26.1MiB/s (27.4MB/s-27.4MB/s), io=3136MiB (3288MB), run=120053-120053msec WRITE: bw=26.3MiB/s (27.6MB/s), 26.3MiB/s-26.3MiB/s (27.6MB/s-27.6MB/s), io=3161MiB (3315MB), run=120053-120053msec without NCQ READ: bw=22.0MiB/s (23.1MB/s), 22.0MiB/s-22.0MiB/s (23.1MB/s-23.1MB/s), io=2647MiB (2775MB), run=120069-120069msec WRITE: bw=22.2MiB/s (23.3MB/s), 22.2MiB/s-22.2MiB/s (23.3MB/s-23.3MB/s), io=2665MiB (2795MB), run=120069-120069msec

Related work items: sonic-net#276, sonic-net#305, sonic-net#332, sonic-net#338, sonic-net#339, sonic-net#1188, sonic-net#1192, sonic-net#1197, sonic-net#1206, sonic-net#1685, sonic-net#1690, sonic-net#1696, sonic-net#1699, sonic-net#1709, sonic-net#1727, sonic-net#1737, sonic-net#1741, sonic-net#1742, sonic-net#2511, sonic-net#2512, sonic-net#2532, sonic-net#2559, sonic-net#2626, sonic-net#2638, sonic-net#2645, sonic-net#2649, sonic-net#2660, sonic-net#2669, sonic-net#2670, sonic-net#2678, sonic-net#10084, sonic-net#11442, sonic-net#11873, sonic-net#12047, sonic-net#12110, sonic-net#12207, sonic-net#12529, sonic-net#12678, sonic-net#13235, sonic-net#13287, sonic-net#13372, sonic-net#13395, sonic-net#13456, sonic-net#13497, sonic-net#13522, sonic-net#13545, sonic-net#13547, sonic-net#13552, sonic-net#13569, sonic-net#13572, sonic-net#13578, sonic-net#13591, sonic-net#13611, sonic-net#13647, sonic-net#13649, sonic-net#13660, sonic-net#13710, sonic-net#13716, sonic-net#13724, sonic-net#13726, sonic-net#13732, sonic-net#13735, sonic-net#13739, sonic-net#13757, sonic-net#13786, sonic-net#13792, sonic-net#13800, sonic-net#13801, sonic-net#13802, sonic-net#13805, sonic-net#13806, sonic-net#13812, sonic-net#13814, sonic-net#13822, sonic-net#13831, sonic-net#13834, sonic-net#13847, sonic-net#13870, sonic-net#13882, sonic-net#13884, sonic-net#13885, sonic-net#13894, sonic-net#13895, sonic-net#13926, sonic-net#13932, sonic-net#13935, sonic-net#13942, sonic-net#13951, sonic-net#13953, sonic-net#13964

prgeor · 2023-03-30T00:01:42Z

@abdosi please help cherry pick to 201911

abdosi · 2023-03-30T00:09:37Z

@Staphylo / @prgeor cherry-pick has conflict. Please create PR for 201911

Why I did it Fix similar issue seen on sonic-net#13739 but only for DCS-7050CX3-32S How I did it Add a kernel parameter to tell libata to disable NCQ How to verify it The message ata2.00: FORCE: horkage modified (noncq) should appear on the dmesg. Test results using: fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 with NCQ READ: bw=26.1MiB/s (27.4MB/s), 26.1MiB/s-26.1MiB/s (27.4MB/s-27.4MB/s), io=3136MiB (3288MB), run=120053-120053msec WRITE: bw=26.3MiB/s (27.6MB/s), 26.3MiB/s-26.3MiB/s (27.6MB/s-27.6MB/s), io=3161MiB (3315MB), run=120053-120053msec without NCQ READ: bw=22.0MiB/s (23.1MB/s), 22.0MiB/s-22.0MiB/s (23.1MB/s-23.1MB/s), io=2647MiB (2775MB), run=120069-120069msec WRITE: bw=22.2MiB/s (23.3MB/s), 22.2MiB/s-22.2MiB/s (23.3MB/s-23.3MB/s), io=2665MiB (2795MB), run=120069-120069msec

Why I did it Fix similar issue seen on #13739 but only for DCS-7050CX3-32S How I did it Add a kernel parameter to tell libata to disable NCQ How to verify it The message ata2.00: FORCE: horkage modified (noncq) should appear on the dmesg. Test results using: fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 with NCQ READ: bw=26.1MiB/s (27.4MB/s), 26.1MiB/s-26.1MiB/s (27.4MB/s-27.4MB/s), io=3136MiB (3288MB), run=120053-120053msec WRITE: bw=26.3MiB/s (27.6MB/s), 26.3MiB/s-26.3MiB/s (27.6MB/s-27.6MB/s), io=3161MiB (3315MB), run=120053-120053msec without NCQ READ: bw=22.0MiB/s (23.1MB/s), 22.0MiB/s-22.0MiB/s (23.1MB/s-23.1MB/s), io=2647MiB (2775MB), run=120069-120069msec WRITE: bw=22.2MiB/s (23.3MB/s), 22.2MiB/s-22.2MiB/s (23.3MB/s-23.3MB/s), io=2665MiB (2795MB), run=120069-120069msec

- Why I did it Based on some research some products might experience an occasional IO failures in the communication between CPU and SSD because of NCQ. There seems to be a problem between some kernel versions and some SATA controllers. Syslog error message examples: Error "ata1: SError: { UnrecovData Handshk }" - "failed command: WRITE FPDMA QUEUED". Error "ata1: SError: { RecovComm HostInt PHYRdyChg CommWake 10B8B DevExch }" - "failed command: READ FPDMA QUEUED". Some vendors already disabled NCQ on their platforms in SONiC due to similar issue: [Arista] Disable ATA NCQ for a few products #13739 [Arista] Disable ATA NCQ for a few products [Arista] Disable SSD NCQ on DCS-7050CX3-32S #13964 [Arista] Disable SSD NCQ on DCS-7050CX3-32S Also there are other discussions on Debian/Ubuntu forums about similar issues and it was suggested to disable NCQ: https://askubuntu.com/questions/133946/are-these-sata-errors-dangerous - How I did it Add a kernel parameter to tell libata to disable NCQ - How to verify it Use FIO tool - fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4

- Why I did it Based on some research some products might experience an occasional IO failures in the communication between CPU and SSD because of NCQ. There seems to be a problem between some kernel versions and some SATA controllers. Syslog error message examples: Error "ata1: SError: { UnrecovData Handshk }" - "failed command: WRITE FPDMA QUEUED". Error "ata1: SError: { RecovComm HostInt PHYRdyChg CommWake 10B8B DevExch }" - "failed command: READ FPDMA QUEUED". Some vendors already disabled NCQ on their platforms in SONiC due to similar issue: [Arista] Disable ATA NCQ for a few products sonic-net#13739 [Arista] Disable ATA NCQ for a few products [Arista] Disable SSD NCQ on DCS-7050CX3-32S sonic-net#13964 [Arista] Disable SSD NCQ on DCS-7050CX3-32S Also there are other discussions on Debian/Ubuntu forums about similar issues and it was suggested to disable NCQ: https://askubuntu.com/questions/133946/are-these-sata-errors-dangerous - How I did it Add a kernel parameter to tell libata to disable NCQ - How to verify it Use FIO tool - fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4

- Why I did it Based on some research some products might experience an occasional IO failures in the communication between CPU and SSD because of NCQ. There seems to be a problem between some kernel versions and some SATA controllers. Syslog error message examples: Error "ata1: SError: { UnrecovData Handshk }" - "failed command: WRITE FPDMA QUEUED". Error "ata1: SError: { RecovComm HostInt PHYRdyChg CommWake 10B8B DevExch }" - "failed command: READ FPDMA QUEUED". Some vendors already disabled NCQ on their platforms in SONiC due to similar issue: [Arista] Disable ATA NCQ for a few products #13739 [Arista] Disable ATA NCQ for a few products [Arista] Disable SSD NCQ on DCS-7050CX3-32S #13964 [Arista] Disable SSD NCQ on DCS-7050CX3-32S Also there are other discussions on Debian/Ubuntu forums about similar issues and it was suggested to disable NCQ: https://askubuntu.com/questions/133946/are-these-sata-errors-dangerous - How I did it Add a kernel parameter to tell libata to disable NCQ - How to verify it Use FIO tool - fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4

Staphylo requested a review from lguohan as a code owner February 9, 2023 15:53

prgeor reviewed Feb 9, 2023

View reviewed changes

files/Aboot/boot0.j2 Show resolved Hide resolved

prgeor added Request for 202012 Branch Request for 202205 Branch Request for 202211 Branch labels Feb 10, 2023

Staphylo added 2 commits February 10, 2023 06:56

Add comment with affected products

2b5e63e

Staphylo force-pushed the master-gardena-ncq branch from 1277127 to 2b5e63e Compare February 10, 2023 14:56

prgeor approved these changes Feb 15, 2023

View reviewed changes

lguohan merged commit 5ce1b8e into sonic-net:master Feb 15, 2023

qiluo-msft added the Included in 202012 Branch label Feb 16, 2023

yxieca added the Approved for 202205 Branch label Feb 17, 2023

mssonicbld added the Created PR to 202205 Branch label Feb 17, 2023

mssonicbld mentioned this pull request Feb 17, 2023

[action] [PR:13739] [Arista] Disable ATA NCQ for a few products #13842

Merged

StormLiangMS added the Approved for 202211 Branch label Feb 17, 2023

mssonicbld added the Created PR to 202211 Branch label Feb 17, 2023

mssonicbld mentioned this pull request Feb 17, 2023

[action] [PR:13739] [Arista] Disable ATA NCQ for a few products #13865

Merged

mssonicbld added a commit that referenced this pull request Feb 17, 2023

[Arista] Disable ATA NCQ for a few products (#13739) (#13865)

1aa3392

mssonicbld added Included in 202211 Branch and removed Approved for 202211 Branch Created PR to 202211 Branch labels Feb 17, 2023

mssonicbld added Included in 202205 Branch and removed Approved for 202205 Branch Created PR to 202205 Branch labels Feb 21, 2023

Staphylo mentioned this pull request Feb 23, 2023

[Arista] Disable SSD NCQ on DCS-7050CX3-32S #13964

Merged

8 tasks

prgeor added the Request for 201911 Branch label Mar 30, 2023

Staphylo mentioned this pull request Mar 30, 2023

[201911][Arista] Disable ATA NCQ for a few products #14468

Merged

volodymyrsamotiy mentioned this pull request Dec 19, 2023

[Mellanox] Disable SSD NCQ on Mellanox platforms #17567

Merged

12 tasks

volodymyrsamotiy mentioned this pull request Jan 3, 2024

[202205] [Mellanox] Disable SSD NCQ on Mellanox platforms #17662

Merged

12 tasks

mssonicbld mentioned this pull request Jan 31, 2024

[action] [PR:17567] [Mellanox] Disable SSD NCQ on Mellanox platforms #17960

Merged

12 tasks

volodymyrsamotiy mentioned this pull request Feb 5, 2024

[202311] [Mellanox] Disable SSD NCQ on Mellanox platforms #18040

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Arista] Disable ATA NCQ for a few products #13739

[Arista] Disable ATA NCQ for a few products #13739

Staphylo commented Feb 9, 2023 •

edited

Loading

Staphylo commented Feb 9, 2023

prgeor commented Feb 10, 2023

Staphylo commented Feb 10, 2023

prgeor commented Feb 15, 2023

prgeor commented Feb 15, 2023

prgeor commented Feb 15, 2023

prgeor commented Feb 15, 2023

mssonicbld commented Feb 17, 2023

mssonicbld commented Feb 17, 2023

prgeor commented Mar 30, 2023

abdosi commented Mar 30, 2023

[Arista] Disable ATA NCQ for a few products #13739

[Arista] Disable ATA NCQ for a few products #13739

Conversation

Staphylo commented Feb 9, 2023 • edited Loading

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Staphylo commented Feb 9, 2023

prgeor commented Feb 10, 2023

Staphylo commented Feb 10, 2023

prgeor commented Feb 15, 2023

prgeor commented Feb 15, 2023

prgeor commented Feb 15, 2023

prgeor commented Feb 15, 2023

mssonicbld commented Feb 17, 2023

mssonicbld commented Feb 17, 2023

prgeor commented Mar 30, 2023

abdosi commented Mar 30, 2023

Staphylo commented Feb 9, 2023 •

edited

Loading