kernel memory corruption __kmalloc, kfree

[fourtrax]

Hi, I trying to find out the reason for a kernel panic on my CM3.
The following kernel panic occurs randomly after a few hours of runtime, most of the times in kmalloc:

[14886.929616] Unable to handle kernel paging request at virtual address 6f616943
[24793.191864 14859.478431] [14886.936863] pgd = a740c000
[24793.193363 0.001499] [14886.939573] [6f616943] *pgd=2adba835, *pte=00000000, *ppte=00000000
[24793.196859 0.003496] [14886.945863] Internal error: Oops: 17 [#1] SMP ARM
[24793.199464 0.002605] [14886.950569] Modules linked in: evdev joydev edt_ft5x06(O) arc4 ecb md4 md5 hmac nls_utf8 cifs s1d13781fb(O) spi_config(O) bnep cfg80211 snd_soc_wm8731 spidev snd_soc_bcm2835_i2s snd_soc_rpi_proto snd_soc_core snd_compress snd_pcm_dmaengine snd_pcm i2c_bcm2835 snd_timer snd spi_bcm2835 bcm2835_gpiomem btusb btrtl btintel btbcm bluetooth rfkill uio_pdrv_genirq uio fixed bcm2835_v4l2 videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core v4l2_common videodev media i2c_dev ipv6
[24793.244143 0.044679] [14886.993678] CPU: 3 PID: 1056 Comm: queue1:src Tainted: GF O 4.9.20-v7+ #985
[24793.246447 0.002304] [14887.001679] Hardware name: BCM2835
[24793.247362 0.000915] [14887.005079] task: aacb49c0 task.stack: a7556000
[24793.250032 0.002670] [14887.009617] PC is at __kmalloc+0x9c/0x274
[24793.264085 0.014053] [14887.013626] LR is at __kmalloc+0x30/0x274
[24793.265160 0.001075] [14887.017636] pc : [<8025dbac>] lr : [<8025db40>] psr: 20000093
[24793.266802 0.001642] [14887.017636] sp : a75578b8 ip : a75578b8 fp : a75578fc
[24793.269737 0.002935] [14887.029112] r10: ad801e40 r9 : 8058ffbc r8 : 02088020
[24793.284257 0.014520] [14887.034335] r7 : 0000002c r6 : acc23418 r5 : ad801e40 r4 : 6f616943
[24793.285919 0.001662] [14887.040860] r3 : 00000000 r2 : 80b81420 r1 : 2e45a000 r0 : a75578b8
[24793.289286 0.003367] [14887.047387] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[24793.304254 0.014968] [14887.054608] Control: 10c5383d Table: 2740c06a DAC: 00000055
[24793.305650 0.001396] [14887.060352] Process queue1:src (pid: 1056, stack limit = 0xa7556210)
[24793.309260 0.003610] [14887.066704] Stack: (0xa75578b8 to 0xa7558000)
[24793.310320 0.001060] [14887.071059] 78a0: ad88dd00 8025db40
[24793.325299 0.014979] [14887.079241] 78c0: 00000003 002f8d29 a755791c a75578d8 8025db40 ad138000 9cc99040 acc23418
[24793.329348 0.004049] [14887.087421] 78e0: 9cc99040 ad13c400 acc233f8 9cc5d680 a755790c a7557900 8058ffbc 8025db1c
[24793.344493 0.015145] [14887.095601] 7900: a7557924 a7557910 8058a5b4 8058ffa4 ad138000 00000001 a7557954 a7557928
[24793.346367 0.001874] [14887.103781] 7920: 80583a14 8058a594 a755797c 80556128 a7557954 00000000 00000000 9cc99040
[24793.363720 0.017353] [14887.111961] 7940: ad138000 ad13c400 a75579ac a7557958 80585780 805839c8 000005ae 00000000
[24793.366893 0.003173] [14887.120140] 7960: 00000000 00000001 00000000 80114234 00000000 acc23418 00000000 60000093
[24793.369763 0.002870] [14887.128321] 7980: 80c02d00 9cc5d680 ad13c400 9cc5d688 00000000 02080020 00000003 40000013
[24793.385084 0.015321] [14887.136504] 79a0: a7557a64 a75579b0 80558080 805855c8 aacb4de4 80b7e940 80c0442c 00000000
[24793.387910 0.002826] [14887.144684] 79c0: a7557a44 a75579d0 80151348 8014f7ac ad803000 80c03180 00164256 00000003
[24793.403928 0.016018] [14887.152866] 79e0: 2224f3b0 00000d8a a7557a04 a75579f8 8014d294 8015a88c a7557a74 a7557a08
[24793.406766 0.002838] [14887.161047] 7a00: 8015a88c 80717bb4 a7557a44 80b7e940 acf13880 ad943b00 aefd8940 80713434
[24793.410003 0.003237] [14887.169228] 7a20: a7557a74 805045e0 ad27688c a7556028 8013e11c 60000093 00000005 9cc5d680
[24793.425370 0.015367] [14887.177408] 7a40: 02080020 00000002 00000000 ad204c00 00000003 40000013 a7557aac a7557a68
[24793.428138 0.002768] [14887.185589] 7a60: 80559be4 80557fc4 a7557a94 a7557a78 805045f0 807180c4 ad276800 00000001
[24793.444196 0.016058] [14887.193769] 7a80: ad135800 ace5dd80 9cc5d680 ad135800 ad135dec ad135d00 00000000 40000013
[24793.446904 0.002708] [14887.201949] 7aa0: a7557ae4 a7557ab0 8054c4c 805598f0 a7557b04 00000000 aacb49c0 ace5dd80
[24793.450320 0.003416] [14887.210129] 7ac0: ad1a1800 ad135800 ace5dd80 000005a2 00000000 ad1a1800 a7557b3c a7557ae8
[24793.465556 0.015236] [14887.218309] 7ae0: 80629ca4 8054c328 a7557b2c 00000000 ad1a1800 a7557b44 80c75ae4 80c02d00
[24793.469073 0.003517] [14887.226489] 7b00: 80c049b8 80c0312c a7556028 00000000 80629ba8 acd2ce00 ad1a1800 acd2ce90
[24793.484496 0.015423] [14887.234669] 7b20: ace5dd80 ad135800 00000003 ad1a1800 a7557b6c a7557b40 80651178 80629be4
[24793.487113 0.002617] [14887.242849] 7b40: 80461490 00000010 1b42ebbd 00000001 1b42e61b 00000001 00000000 acd2ce00
[24793.490585 0.003472] [14887.251029] 7b60: a7557bc4 a7557b70 8062a460 80651094 acd2ce90 00000001 000005a2 00000000
[24793.505918 0.015333] [14887.259209] 7b80: 00000000 acd2ce90 a7557ba4 fffffff4 00000000 8046125c a7557be4 a75452a0
[24793.509236 0.003318] [14887.267389] 7ba0: a7545200 00000002 0000000e 80c64d80 000005dc ace5dd80 a7557bd4 a7557bc8
[24793.524640 0.015404] [14887.275569] 7bc0: 8062a6ac 80629f2c a7557c0c a7557bd8 8066b934 8062a69c a7557c04 1bbca8c0
[24793.527094 0.002454] [14887.283749] 7be0: 8066d25c ace5dd80 80c64d80 aaf958c0 00000000 80c64d80 000005dc 1bbca8c0
[24793.543565 0.016471] [14887.291929] 7c00: a7557c44 a7557c10 8066d568 8066b7b4 8066e018 000d34a8 00000000 ace5dd80
[24793.545965 0.002400] [14887.300109] 7c20: 80c64d80 000d34a8 00000000 aaf958c0 ad135800 1bbca8c0 a7557c94 a7557c48
[24793.549470 0.003505] [14887.308289] 7c40: 8066e0dc 8066d43c 8066d70c 8066b740 00000000 80c64d80 8066c6bc 00000000
[24793.565071 0.015601] [14887.316469] 7c60: aaf958c0 a7557d50 00000580 8066d1bc ace5dd80 aaf958c0 80c64d80 ad1d6834
[24793.567415 0.002344] [14887.324650] 7c80: 00000000 a7557d50 a7557cb4 a7557c98 8066d7f4 8066e024 000017ff ace5dd80
[24793.583739 0.016324] [14887.332832] 7ca0: 80c64d80 00000580 a7557ccc a7557cb8 8066eab8 8066d7b8 ace5dd80 aaf958c0
[24793.586053 0.002314] [14887.341014] 7cc0: a7557cf4 a7557cd0 80698360 8066eaa0 aaf958c0 ace5dd80 a7557f4c 00000578
[24793.589783 0.003730] [14887.349195] 7ce0: a7557d50 00000000 a7557dec a7557cf8 80699d50 80698148 00000580 00000008
[24793.605035 0.015252] [14887.357375] 7d00: a7557d34 a7557d30 00004040 0000001c 80c64d80 00000000 00008e13 8066d1bc
[24793.606467 0.001432] [14887.365556] 7d20: 00000000 00000000 aaf95b10 00000000 00000000 a7557e1c 1f330000 1bbca8c0
[24793.623753 0.017286] [14887.373736] 7d40: 00000000 00000000 ffff0000 00000d8a 00000002 00000001 00000000 00110000
[24793.624979 0.001226] [14887.381916] 7d60: 00000000 a7557e48 00000000 00000000 19bca8c0 1bbca8c0 62858e13 a7557d88
[24793.629553 0.004574] [14887.390097] 7d80: 80271450 804614f0 a7557f54 00000000 a7557e24 00000001 00000003 00004000
[24793.644567 0.015014] [14887.398277] 7da0: a7557de4 a7557db0 804695e8 802713e4 a7557e48 a7557dbc 00000010 a7557e48
[24793.649061 0.004494] [14887.406457] 7dc0: a7557de4 aaf958c0 00000000 ac1a6fc0 00000000 00000000 00004000 a7557e28
[24793.663947 0.014886] [14887.414637] 7de0: a7557e0c a7557df0 806a3f90 80699ae4 a7557e24 a7557e24 a7557f4c 00000000
[24793.665180 0.001233] [14887.422817] 7e00: a7557e1c a7557e10 806098a8 806a3f00 a7557f34 a7557e20 80609f88 80609890
[24793.669587 0.004407] [14887.430997] 7e20: acf13880 00000000 00000000 00000000 00000000 ffffffff 60000013 ffffffff
[24793.684703 0.015116] [14887.439177] 7e40: a7557eac 80713434 6f9676a0 0000000c 6f963218 00000002 6f84b07f 0000056a
[24793.689180 0.004477] [14887.447358] 7e60: 024080c0 ad801cc0 a7557e88 ad60d518 ad60d5f0 9b0aa490 a7557eb4 a7557e88
[24793.704066 0.014886] [14887.455537] 7e80: 8028cdb4 80286af8 8e130002 1bbca8c0 00000000 00000000 00000000 9b0aa490
[24793.705299 0.001233] [14887.463717] 7ea0: 9b0aa440 ad60d518 ad60d518 00000008 a7557ed4 a7557ec0 80286af8 8025eb4c
[24793.723507 0.018208] [14887.471898] 7ec0: 00000003 9b0aa440 a7557ef4 a7557ed8 80286cb8 8028f57c acc17b00 a7557f48
[24793.724722 0.001215] [14887.480077] 7ee0: a7557f44 68ba04b8 00000128 80108244 a7557f0c a7557f00 8028f57c 8028ed38
[24793.729276 0.004554] [14887.488257] 7f00: a7557f1c a7557f10 8028f5e0 ac1a6fc0 00004000 68ba04b8 00000128 80108244
[24793.744304 0.015028] [14887.496437] 7f20: a7556000 00000000 a7557f94 a7557f38 8060acfc 80609dc8 00000000 00000000
[24793.745567 0.001263] [14887.504617] 7f40: aacb49c0 00000001 fffffff7 a7557e88 00000010 00000001 00000000 00000000
[24793.763582 0.018015] [14887.512798] 7f60: a7557e60 00000000 00000000 00000000 00004040 00000000 68ba04b8 00004000
[24793.764811 0.001229] [14887.520980] 7f80: 68ba04b8 6ac0e200 a7557fa4 a7557f98 8060ad44 8060acb8 00000000 a7557fa8
[24793.769388 0.004577] [14887.529162] 7fa0: 801080c0 8060ad38 00004000 68ba04b8 0000001d 68ba04b8 00004000 6ac0e118
[24793.784293 0.014905] [14887.537343] 7fc0: 00004000 68ba04b8 6ac0e200 00000128 68ba0550 725f4728 7364b8d0 68ba04fc
[24793.785543 0.001250] [14887.545524] 7fe0: 00000000 68ba0470 75cc3504 75cc43a4 80000010 0000001d 00000000 00000000
[24793.803805 0.018262] [14887.553726] [<8025dbac>] (__kmalloc) from [<8058ffbc>] (__DWC_ALLOC_ATOMIC+0x24/0x28)
[24793.804983 0.001178] [14887.561574] [<8058ffbc>] (__DWC_ALLOC_ATOMIC) from [<8058a5b4>] (dwc_otg_hcd_qtd_create+0x2c/0x60)
[24793.809712 0.004729] [14887.570543] [<8058a5b4>] (dwc_otg_hcd_qtd_create) from [<80583a14>] (dwc_otg_hcd_urb_enqueue+0x58/0x204)
[24793.824757 0.015045] [14887.580033] [<80583a14>] (dwc_otg_hcd_urb_enqueue) from [<80585780>] (dwc_otg_urb_enqueue+0x1c4/0x354)
[24793.829313 0.004556] [14887.589349] [<80585780>] (dwc_otg_urb_enqueue) from [<80558080>] (usb_hcd_submit_urb+0xc8/0x950)
[24793.844399 0.015086] [14887.598143] [<80558080>] (usb_hcd_submit_urb) from [<80559be4>] (usb_submit_urb+0x300/0x4bc)
[24793.848937 0.004538] [14887.606593] [<80559be4>] (usb_submit_urb) from [<8054c4c4>] (usbnet_start_xmit+0x1a8/0x58c)
[24793.863965 0.015028] [14887.614956] [<8054c4c4>] (usbnet_start_xmit) from [<80629ca4>] (dev_hard_start_xmit+0xcc/0x244)
[24793.865278 0.001313] [14887.623662] [<80629ca4>] (dev_hard_start_xmit) from [<80651178>] (sch_direct_xmit+0xf0/0x1c4)
[24793.883641 0.018363] [14887.632193] [<80651178>] (sch_direct_xmit) from [<8062a460>] (__dev_queue_xmit+0x540/0x770)
[24793.884912 0.001271] [14887.640549] [<8062a460>] (__dev_queue_xmit) from [<8062a6ac>] (dev_queue_xmit+0x1c/0x20)
[24793.889314 0.004402] [14887.648648] [<8062a6ac>] (dev_queue_xmit) from [<8066b934>] (ip_finish_output2+0x18c/0x3ac)
[24793.904239 0.014925] [14887.657007] [<8066b934>] (ip_finish_output2) from [<8066d568>] (ip_finish_output+0x138/0x200)
[24793.905556 0.001317] [14887.665538] [<8066d568>] (ip_finish_output) from [<8066e0dc>] (ip_output+0xc4/0x138)
[24793.923718 0.018162] [14887.673288] [<8066e0dc>] (ip_output) from [<8066d7f4>] (ip_local_out+0x48/0x4c)
[24793.924820 0.001102] [14887.680604] [<8066d7f4>] (ip_local_out) from [<8066eab8>] (ip_send_skb+0x24/0xa4)
[24793.929094 0.004274] [14887.688094] [<8066eab8>] (ip_send_skb) from [<80698360>] (udp_send_skb+0x224/0x2d8)
[24793.944106 0.015012] [14887.695757] [<80698360>] (udp_send_skb) from [<80699d50>] (udp_sendmsg+0x278/0x7f8)
[24793.945247 0.001141] [14887.703421] [<80699d50>] (udp_sendmsg) from [<806a3f90>] (inet_sendmsg+0x9c/0xd0)
[24793.949686 0.004439] [14887.710914] [<806a3f90>] (inet_sendmsg) from [<806098a8>] (sock_sendmsg+0x24/0x34)
[24793.964510 0.014824] [14887.718491] [<806098a8>] (sock_sendmsg) from [<80609f88>] (___sys_sendmsg+0x1cc/0x1f4)
[24793.968884 0.004374] [14887.726415] [<80609f88>] (___sys_sendmsg) from [<8060acfc>] (__sys_sendmsg+0x50/0x80)
[24793.983919 0.015035] [14887.734251] [<8060acfc>] (__sys_sendmsg) from [<8060ad44>] (SyS_sendmsg+0x18/0x1c)
[24793.985074 0.001155] [14887.741831] [<8060ad44>] (SyS_sendmsg) from [<801080c0>] (ret_fast_syscall+0x0/0x1c)
[24793.989318 0.004244] [14887.749581] Code: e7914002 e3540000 0a000065 e5953014 (e7943003)
[24794.004062 0.014744] [14887.755680] ---[ end trace 468f5b0bc9e4c731 ]---
[24794.004766 0.000704] [14887.760298] Kernel panic - not syncing: Fatal exception in interrupt
[24794.009007 0.004241] [14887.766663] CPU1: stopping
[24794.009435 0.000428] [14887.769386] CPU: 1 PID: 1058 Comm: queue2:src Tainted: GF D O 4.9.20-v7+ #985
[24794.024305 0.014870] [14887.777386] Hardware name: BCM2835
[24794.024815 0.000510] [14887.780809] [<8010fb3c>] (unwind_backtrace) from [<8010c058>] (show_stack+0x20/0x24)
[24794.029213 0.004398] [14887.788562] [<8010c058>] (show_stack) from [<80454fa0>] (dump_stack+0xd4/0x118)
[24794.044103 0.014890] [14887.795881] [<80454fa0>] (dump_stack) from [<8010e1e4>] (handle_IPI+0x2a4/0x2c4)
[24794.045242 0.001139] [14887.803284] [<8010e1e4>] (handle_IPI) from [<801014e0>] (bcm2836_arm_irqchip_handle_irq+0x7c/0xac)
[24794.063574 0.018332] [14887.812253] [<801014e0>] (bcm2836_arm_irqchip_handle_irq) from [<8071874c>] (__irq_usr+0x4c/0x60)
[24794.064917 0.001343] [14887.821124] Exception stack(0xacedbfb0 to 0xacedbff8)
[24794.068661 0.003744] [14887.826176] bfa0: 72440084 72440124 724400e4 72440104
[24794.083875 0.015214] [14887.834356] bfc0: 72440164 724400a4 c18e3094 41159c36 c1585806 c0e7cfc5 724400a0 413d2168
[24794.085121 0.001246] [14887.842534] bfe0: 724400c4 67ba0218 72440144 6bb44208 80000010 ffffffff
[24795.244483 1.159362] [14888.984533] SMP: failed to stop secondary CPUs
[24795.246261 0.001778] [14888.988979] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

Also in kfree once:

[ 1912.478596] Unable to handle kernel paging request at virtual address aff0db2c
[2638.735024 1891.095338] [ 1912.485901] pgd = 80004000
[2638.736429 0.001405] [ 1912.488612] [aff0db2c] *pgd=2dba0811, *pte=00000000, *ppte=00000000
[2638.738047 0.001618] [ 1912.494950] Internal error: Oops: 7 [#1] SMP ARM
[2638.747787 0.009740] [ 1912.499565] Modules linked in: s1d13781fb(O) spi_config(O) evdev joydev edt_ft5x06(O) bnep snd_soc_wm8731 snd_soc_bcm2835_i2s snd_soc_rpi_proto snd_soc_core spidev snd_compress snd_pcm_dmaengine snd_pcm snd_timer bcm2835_gpiomem snd i2c_bcm2835 spi_bcm2835 8723bu(O) btusb cfg80211 btrtl btintel btbcm bluetooth rfkill uio_pdrv_genirq uio fixed bcm2835_v4l2 videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core v4l2_common videodev media i2c_dev ipv6
[2638.787729 0.039942] [ 1912.540379] CPU: 1 PID: 81 Comm: VCHIQ-0 Tainted: GF O 4.9.20-v7+ #985
[2638.789032 0.001303] [ 1912.547946] Hardware name: BCM2835
[2638.789669 0.000637] [ 1912.551345] task: ad142c40 task.stack: ad12c000
[2638.806843 0.017174] [ 1912.555881] PC is at kfree+0x48/0x1e8
[2638.807444 0.000601] [ 1912.559547] LR is at vchiq_complete_bulk+0x2b0/0x354
[2638.808223 0.000779] [ 1912.564511] pc : [<8025f168>] lr : [<80603078>] psr: 20000013
[2638.809206 0.000983] [ 1912.564511] sp : ad12de60 ip : 0086a6c6 fp : ad12de8c
[2638.826869 0.017663] [ 1912.575988] r10: 6f616967 r9 : 80c06984 r8 : 80d03818
[2638.827800 0.000931] [ 1912.581211] r7 : 00000000 r6 : 6f616943 r5 : 80603078 r4 : 6f616943
[2638.828824 0.001024] [ 1912.587736] r3 : aff0db18 r2 : ae2973f8 r1 : 00000001 r0 : 6f616943
[2638.830705 0.001881] [ 1912.594264] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[2638.847724 0.017019] [ 1912.601398] Control: 10c5383d Table: 298dc06a DAC: 00000055
[2638.848626 0.000902] [ 1912.607144] Process VCHIQ-0 (pid: 81, stack limit = 0xad12c210)
[2638.850313 0.001687] [ 1912.613061] Stack: (0xad12de60 to 0xad12e000)
[2638.867022 0.016709] [ 1912.617424] de60: ad12de8c 80218148 6f616943 ae2973e8 00000000 00000000 80d03818 80c06984
[2638.868320 0.001298] [ 1912.625606] de80: ad12ded4 ad12de90 80603078 8025f12c 8012d160 801e2fdc 0000000f 6f61695b
[2638.870505 0.002185] [ 1912.633787] dea0: ae5c1080 6f616967 ad12ded4 00000010 00000008 a97b98dc 80c64a34 a97b9800
[2638.887890 0.017385] [ 1912.641969] dec0: ae596648 80d1489c ad12df5c ad12ded8 805fc65c 80602dd4 80713434 80144600
[2638.889230 0.001340] [ 1912.650153] dee0: 8014834c 80147f18 ad12df24 ad12def8 80160c84 8014833c fff9fee9 a97b9950
[2638.907210 0.017980] [ 1912.658334] df00: 80845d58 ae580194 ae580020 a97b9990 01af6658 0000000b 00000009 ae580194
[2638.908487 0.001277] [ 1912.666515] df20: fff9fee9 ffffffff 00000000 00000000 00000000 00000000 ad05f6c0 80d1489c
[2638.910660 0.002173] [ 1912.674697] df40: 805fb4e0 00000000 00000000 00000000 ad12dfac ad12df60 8013c7b0 805fb4ec
[2638.927970 0.017310] [ 1912.682878] df60: 2e43e000 00000000 54155145 80d1489c 00000000 00000000 ad12df78 ad12df78
[2638.929311 0.001341] [ 1912.691059] df80: 00000000 00000000 ad12df88 ad12df88 ad05f6c0 8013c6c4 00000000 00000000
[2638.947270 0.017959] [ 1912.699241] dfa0: 00000000 ad12dfb0 80108148 8013c6d0 00000000 00000000 00000000 00000000
[2638.948559 0.001289] [ 1912.707422] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[2638.950771 0.002212] [ 1912.715601] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 55757575 55d5d55f
[2638.968180 0.017409] [ 1912.723798] [<8025f168>] (kfree) from [<80603078>] (vchiq_complete_bulk+0x2b0/0x354)
[2638.969381 0.001201] [ 1912.731554] [<80603078>] (vchiq_complete_bulk) from [<805fc65c>] (slot_handler_func+0x117c/0x1858)
[2638.987570 0.018189] [ 1912.740522] [<805fc65c>] (slot_handler_func) from [<8013c7b0>] (kthread+0xec/0x104)
[2638.988755 0.001185] [ 1912.748187] [<8013c7b0>] (kthread) from [<80108148>] (ret_from_fork+0x14/0x2c)
[2638.990718 0.001963] [ 1912.755414] Code: e1a0c62c e5933000 e08cc18c e083310c (e5931014)
[2639.007559 0.016841] [ 1912.761623] ---[ end trace 09762ec6fae8956c ]---

To debug this I did insert "slub_debug" parameter to the kernel and ran "./slabinfo -v", which shows me this:

[ 119.944481] =============================================================================
[ 119.944493] BUG kmalloc-512 (Not tainted): Redzone overwritten
[ 119.944496] -----------------------------------------------------------------------------

[ 119.944498] Disabling lock debugging due to kernel taint
[ 119.944504] INFO: 0xadb51e08-0xadb51e0c. First byte 0x40 instead of 0xcc
[ 119.944517] INFO: Allocated in platform_device_alloc+0x34/0x78 age=11951 cpu=0 pid=1
[ 119.944523] __slab_alloc.constprop.8+0x44/0x90
[ 119.944527] __kmalloc+0x240/0x274
[ 119.944531] platform_device_alloc+0x34/0x78
[ 119.944536] of_device_alloc+0x34/0x184
[ 119.944541] of_platform_device_create_pdata+0x58/0xc8
[ 119.944545] of_platform_bus_create+0x110/0x374
[ 119.944550] of_platform_bus_create+0x16c/0x374
[ 119.944554] of_platform_populate+0x7c/0xdc
[ 119.944561] of_platform_default_populate_init+0x74/0x88
[ 119.944566] do_one_initcall+0x50/0x184
[ 119.944571] kernel_init_freeable+0x228/0x2c8
[ 119.944577] kernel_init+0x18/0x124
[ 119.944582] ret_from_fork+0x14/0x2c
[ 119.944587] INFO: Slab 0xae3d1740 objects=21 used=21 fp=0x (null) flags=0x4081
[ 119.944590] INFO: Object 0xadb51e40 @offset=7744 fp=0xadb52140

[ 119.944597] Redzone adb51e00: cc cc cc cc cc cc cc cc 40 cc cc cc 80 cc cc cc ........@.......
[ 119.944601] Redzone adb51e10: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 119.944604] Redzone adb51e20: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 119.944607] Redzone adb51e30: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 119.944611] Object adb51e40: 80 35 bb ad ff ff ff ff 00 00 00 00 00 00 00 00 .5..............
[ 119.944614] Object adb51e50: 50 0f b5 ad c0 e7 ba ad 80 35 bb ad 8c 24 82 ad P........5...$..
[ 119.944617] Object adb51e60: b4 4b c1 ac 58 0f b5 ad c0 a7 9f ad c8 4c c5 80 .K..X........L..
[ 119.944620] Object adb51e70: e8 7c bb ad 07 00 00 00 07 00 00 00 00 00 00 00 .|..............
[ 119.944624] Object adb51e80: 00 00 00 00 01 00 00 00 00 00 00 00 8c 1e b5 ad ................
[ 119.944627] Object adb51e90: 8c 1e b5 ad 00 00 00 00 00 00 00 00 c8 4f c5 80 .............O..
[ 119.944630] Object adb51ea0: 80 c8 c5 80 00 00 00 00 00 9d de ac 00 00 00 00 ................
[ 119.944633] Object adb51eb0: 40 00 00 00 0d 00 0d 00 00 00 00 00 00 00 00 00 @...............
[ 119.944637] Object adb51ec0: 00 00 00 00 00 00 00 00 54 52 50 80 50 1e b5 ad ........TRP.P...
[ 119.944640] Object adb51ed0: 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00 ................
[ 119.944643] Object adb51ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 119.944646] Object adb51ef0: e0 ff ff ff f4 1e b5 ad f4 1e b5 ad d0 52 50 80 .............RP.
[ 119.944650] Object adb51f00: 00 00 00 00 04 1f b5 ad 04 1f b5 ad 00 00 00 00 ................
[ 119.944653] Object adb51f10: 00 00 00 00 01 00 00 00 81 20 00 00 00 00 00 00 ......... ......
[ 119.944656] Object adb51f20: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 119.944659] Object adb51f30: 00 00 00 00 00 00 00 00 fa 8a ff ff 40 85 de ac ............@...
[ 119.944663] Object adb51f40: 00 00 00 00 c0 ea d9 ac e8 c9 3d ad 00 00 00 00 ..........=.....
[ 119.944666] Object adb51f50: 58 1f b5 ad 00 00 00 00 ff ff ff ff 00 00 00 00 X...............
[ 119.944669] Object adb51f60: 00 00 f4 ff 00 00 00 00 fc e7 3b ad fc f9 3b ad ..........;...;.
[ 119.944672] Object adb51f70: 00 00 00 00 00 00 00 00 f4 92 c0 80 00 00 00 00 ................
[ 119.944676] Object adb51f80: f8 5e fe ae 08 5f fe ae 00 00 00 00 00 00 00 00 .^..._..........
[ 119.944679] Object adb51f90: 02 00 02 00 94 1f b5 ad 94 1f b5 ad 00 00 00 00 ................
[ 119.944683] Object adb51fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 119.944686] Object adb51fb0: 00 00 00 00 74 be 4f 80 00 00 00 00 00 00 00 00 ....t.O.........
[ 119.944689] Object adb51fc0: 00 00 00 00 00 00 00 00 04 00 00 00 40 f8 ba ad ............@...
[ 119.944692] Object adb51fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 119.944696] Object adb51fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 119.944699] Object adb51ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 119.944702] Object adb52000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 119.944705] Object adb52010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 119.944709] Object adb52020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 119.944712] Object adb52030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 119.944715] Redzone adb52040: cc cc cc cc ....
[ 119.944719] Padding adb520e8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 119.944722] Padding adb520f8: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
[ 119.944728] CPU: 1 PID: 764 Comm: slabinfo Tainted: G B 4.9.28-v7+ #998
[ 119.944731] Hardware name: BCM2835
[ 119.944742] [<8010fb3c>] (unwind_backtrace) from [<8010c058>] (show_stack+0x20/0x24)
[ 119.944751] [<8010c058>] (show_stack) from [<804552c0>] (dump_stack+0xd4/0x118)
[ 119.944759] [<804552c0>] (dump_stack) from [<8025b338>] (print_trailer+0x184/0x268)
[ 119.944766] [<8025b338>] (print_trailer) from [<8025b4f8>] (check_bytes_and_report+0xdc/0x118)
[ 119.944772] [<8025b4f8>] (check_bytes_and_report) from [<8025b6f8>] (check_object+0x1c4/0x2dc)
[ 119.944778] [<8025b6f8>] (check_object) from [<8025c690>] (validate_slab_slab+0x20c/0x274)
[ 119.944785] [<8025c690>] (validate_slab_slab) from [<8025febc>] (validate_store+0x110/0x19c)
[ 119.944793] [<8025febc>] (validate_store) from [<8025a5a8>] (slab_attr_store+0x44/0xe8)
[ 119.944803] [<8025a5a8>] (slab_attr_store) from [<802ec0d0>] (sysfs_kf_write+0x54/0x58)
[ 119.944811] [<802ec0d0>] (sysfs_kf_write) from [<802eb83c>] (kernfs_fop_write+0xe4/0x1ec)
[ 119.944820] [<802eb83c>] (kernfs_fop_write) from [<8026f328>] (__vfs_write+0x38/0x128)
[ 119.944827] [<8026f328>] (__vfs_write) from [<8027022c>] (vfs_write+0xb4/0x1bc)
[ 119.944833] [<8027022c>] (vfs_write) from [<80271334>] (SyS_write+0x54/0xb0)
[ 119.944841] [<80271334>] (SyS_write) from [<801080c0>] (ret_fast_syscall+0x0/0x1c)
[ 119.944845] FIX kmalloc-512: Restoring 0xadb51e08-0xadb51e0c=0xcc

I am not an expert on kernel debugging but this seems to be related to the kernel panics.

I have tried a fresh installation of 2017-04-10-raspbian-jessie-lite without any modifications and slabinfo shows the same memory corruption, so it doesn't seem to be related to anything I am running on the CM3.

Any input on how to debug this is highly appreciated.
thanks in advance

[pelwell]

Memory corruptions like this are usually the result of an undervoltage situation.

1. How are you powering your CM3?
2. Is it in the CMIO board? Which version?
3. Try running vcgencmd get_throttled and verifying that the result is 0.

N.B. The original CMIO board (V1.x) isn't capable of adequately powering a CM3 under heavy load - you need a V3 board.

[fourtrax]

thanks for you help. You are probably right that it is a hardware problem. What surprises me a little though is that the overwritten bytes reported by slabinfo are always identical.

1. I tested powering it via a 5V/2A wall and a 5A bench supply set to 5V, that made no difference
2. It is a CMIO v1.2 board . The kernel panics happened under load but the reported corruption by slabinfo is there right after start up without any load.
3. vcgencmd get_throttled returns 0 for some time and then returns throttled=0x20002
   The 0x20002 seems to indicate a thermal problem.

I will test with a V3 board and better cooling.

[popcornmix]

I don't think CM3 will report under-voltage through get_throttled.
You'll probably need to use a voltmeter/scope when the processor is under stress to confirm if there is a voltage drop.

[JamesH65]

@fourtrax Can this be closed?

[JamesH65]

This issue will be closed within 30 days unless further interactions are posted. If you wish this issue to remain open, please add a comment. A closed issue may be reopened if requested.

[JamesH65]

Closing due to lack of activity. Please request to be reopened if you feel this issue is still relevant.