Fix D-Cache invalidation at boot and during MPU setup on Cortex-M7

[aurel32]

This PR fixes issues introduced by d6b5023 and reported as #35089, #35107 and #35220.

It's not clear to me if we should check the status of the D-Cache, or if it is ensured to be disabled at reset, so I took the safe way.

[galak]

LGTM. Would like to get validation from @mnkp and @erwango / @ABOSTM that this fixes the 2 issues on STM32 as well before we merge this.

[ioannisg]

Will test this on SAM E70, hang on

Confirmed, this makes the SAM E70 no crash, thanks @aurel32 for the fix

[mnkp]

It would be good to test it on an NXP chip. They use a different MPU.

[galak]

It would be good to test it on an NXP chip. They use a different MPU.

Agreed, will ask @MaureenHelm to try this out. @mnkp does this address the issue you are seeing on Atmel SAM?

[mnkp]

@mnkp does this address the issue you are seeing on Atmel SAM?

Sorry, I didn't mention since @ioannisg has already reported the test result. I've also tested the PR on SAM E70 and it fixes #35107.

[aurel32]

It would be good to test it on an NXP chip. They use a different MPU.

Agreed, will ask @MaureenHelm to try this out. @mnkp does this address the issue you are seeing on Atmel SAM?

I think more testing is definitely welcome. Note however that at the end the issue is not linked with the MPU, but just related to the fact that the SCB_CleanInvalidateDCache() is now called earlier when the cache is not yet enabled.

[MaureenHelm]

I tested a few networking samples with mimxrt1060_evk and not seeing any issues.

The i.MX RT family (CM7) has the standard Arm MPU. It's only the Kinetis family (CM4) that has the NXP MPU.

[ABOSTM]

I confirm issue #35089 is fixed by this PR.
But I could not reproduce the #35220, test is always successful even before this PR,
so I cannot confirm this fixes the issue.
@FRASTM can you check on your side ?

[galak]

@FRASTM any updates on testing this?

[yashi]

Why do we need to clean the d-cache in z_arm_mpu_init()? What data in no-cache area are we trying to write back to the memory? If there is nothing we need to keep, can we just call SCB_InvalidateDCache()?

Just for me to understand the context:

• CONFIG_NOCACHE_MEMORY means we have uncached region in the system memory.
• CONFIG_INIT_ARCH_HW_AT_BOOT means force Zephyr to initialize the hardware to reset state. For this case, we explicitly ask Zephyr not to initialize the hardware (which implicitly means that hardware is clean and initialized by PoR, no?)
• The D-Cache is enabled at init_dcache() at PRE_KERNEL_1 after z_arm_mpu_init().
• The commit ac3d943 added CB_CleanInvalidateDCache() in the function by @CristXu .

[CristXu]

Why do we need to clean the d-cache in z_arm_mpu_init()? What data in no-cache area are we trying to write back to the memory? If there is nothing we need to keep, can we just call SCB_InvalidateDCache()?

Just for me to understand the context:

• CONFIG_NOCACHE_MEMORY means we have uncached region in the system memory.
• CONFIG_INIT_ARCH_HW_AT_BOOT means force Zephyr to initialize the hardware to reset state. For this case, we explicitly ask Zephyr not to initialize the hardware (which implicitly means that hardware is clean and initialized by PoR, no?)
• The D-Cache is enabled at init_dcache() at PRE_KERNEL_1 after z_arm_mpu_init().
• The commit ac3d943 added CB_CleanInvalidateDCache() in the function by @CristXu .

Hi，
This story is from here, usb: mcux RT1060 EVK - when using on-chip memory, USB fails, the ocram which means on-chip memory, the ocram is cachable when boot, and so that , some global varibles is in the ocram already. But after boot, we will configure the mpu, and we place the non-cache area on the ocram, then if we only invalidate the dcache, the variable will gone, then the usb will fail. so we need to clean&invalidate the ocram first, then configure it as non-cache area to save the old variable.

Regards,
Crist

[yashi]

Hi @CristXu,

Thank you for your insight! 😃 Let me clarify a bit.

• Before Zephyr boots
  • you have some values in OCRAM, which Zephyr needs.
  • OCRAM is cacheable at the time.
  • Because OCRAM is cacheable, the values might only be in the D-Cache and not in OCRAM.
• After Zephyr is booted
  • OCRAM is configured be non-cacheable
  • But the USB needs the values in OCRAM
• Thus,
  • you need to make sure the values are in OCRAM, not in the D-Cache.

My question to @CristXu is

• Is it possible to clean/flash the cache to OCRAM before booting Zephyr?

Another question for Zephyr community is

• Is it Zephyr's responsibility to clean/flash the contents of D-cache while booting? What happen if a board with CONFIG_NOCACHE_MEORY=y gets here from a bootloader which made D-cache dirty?

Don't take me wrong. I'm not against anything. I just want to know the protocol / requirements / expectations for booting. In fact, I think it's a good idea to have a region of memory for passing data. But in this case, the data is passed by the cache, not the OCRAM, AFAICS.

[CristXu]

Hi @CristXu,

Thank you for your insight! 😃 Let me clarify a bit.

• Before Zephyr boots

  • you have some values in OCRAM, which Zephyr needs.
  • OCRAM is cacheable at the time.
  • Because OCRAM is cacheable, the values might only be in the D-Cache and not in OCRAM.

• After Zephyr is booted

  • OCRAM is configured be non-cacheable
  • But the USB needs the values in OCRAM

• Thus,

  • you need to make sure the values are in OCRAM, not in the D-Cache.

My question to @CristXu is

• Is it possible to clean/flash the cache to OCRAM before booting Zephyr?

Another question for Zephyr community is

• Is it Zephyr's responsibility to clean/flash the contents of D-cache while booting? What happen if a board with CONFIG_NOCACHE_MEORY=y gets here from a bootloader which made D-cache dirty?

Don't take me wrong. I'm not against anything. I just want to know the protocol / requirements / expectations for booting. In fact, I think it's a good idea to have a region of memory for passing data. But in this case, the data is passed by the cache, not the OCRAM, AFAICS.

Yes, i agree with you. the values need update to the OCRAM after configure this area as the non-cacheable area. But sorry, i do not have any idea of clean/flash the cache to OCRAM before booting zephyr.

[FRASTM]

The pb of the #35220 is not fixed with this PR. The
Because the the soc/arm/st_stm32/stm32f7 does not define the CONFIG_INIT_ARCH_HW_AT_BOOT
then the Cache is not disabled here in the arch/arm/core/aarch32/cortex_m/scb.c

[galak]

@FRASTM would it make sense for stm32f7 to define CONFIG_INIT_ARCH_HW_AT_BOOT?

[FRASTM]

I do not really know what are the impacts of setting the CONFIG_INIT_ARCH_HW_AT_BOOT for the soc/stm32f7 family
especially on the boot phase

[aurel32]

The pb of the #35220 is not fixed with this PR. The

Note that on my side I was able to reproduce the issue from #35220 on a stm32f723e discovery board, and that patch was fixing the problem. That's actually how I produced that patch.

Because the the soc/arm/st_stm32/stm32f7 does not define the CONFIG_INIT_ARCH_HW_AT_BOOT
then the Cache is not disabled here in the arch/arm/core/aarch32/cortex_m/scb.c

The PR is composed of two patches. One fixes the issue in arch/arm/core/aarch32/mpu/arm_mpu.c. This is the originally reported issue.

The other one fixes the issue for CONFIG_INIT_ARCH_HW_AT_BOOT, because i realized that it suffers from exactly the same issue.

Therefore there is no need to select this option for the STM32F7, instead we should understand why the fix doesn't work in your case. For me the bus fault happened in the SCB_CleanInvalidateDCache() function called from arch/arm/core/aarch32/mpu/arm_mpu.c. After this PR, this function should not be called anymore, if the cache is disabled. Could you please check where the bus fault happens on your case? That might help to understand the problem.