Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure to upload filesystem image to ESP32-S3 (ESPTOOL-538) #795

Closed
1 task done
CoryCharlton opened this issue Nov 10, 2022 · 19 comments
Closed
1 task done

Failure to upload filesystem image to ESP32-S3 (ESPTOOL-538) #795

CoryCharlton opened this issue Nov 10, 2022 · 19 comments

Comments

@CoryCharlton
Copy link

Operating System

Windows 10

Esptool Version

v4.2.1 and v4.3

Python Version

Python 3.9.10

Chip Description

ESP32-S3

Device Description

Unexpected Maker FeatherS3 and TinyS3 (another user also reported the same issue with the ProS3): https://esp32s3.com/index.html#esp32s3_info

Hardware Configuration

Nothing is connected

How is Esptool Run

Windows terminal, PlatformIO (VS Code)

Full Esptool Command Line that Was Run

C:\Users\ccharlton\Downloads\esptool-v4.3-win64\esptool-v4.3-win64\esptool.exe --chip esp32s3 --port "COM10" --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size 16MB 13172736 .pio\build\debug\spiffs.bin

Esptool Output

esptool.py v4.3
Serial port COM10
Connecting....
Chip is ESP32-S3 (revision v0.1)
Features: WiFi, BLE
Crystal is 40MHz
MAC: f4:12:fa:42:03:2c
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00c90000 to 0x00ffffff...
Compressed 3604480 bytes to 6617...
Writing at 0x00c90000... (100 %)

A fatal error occurred: Invalid head of packet (0x6F): Possible serial noise or corruption.

More Information

No response

Other Steps to Reproduce

No response

I Have Read the Troubleshooting Guide

  • I confirm I have read the troubleshooting guide.
@github-actions github-actions bot changed the title Failure to upload filesystem image to ESP32-S3 Failure to upload filesystem image to ESP32-S3 (ESPTOOL-538) Nov 10, 2022
@dobairoland
Copy link
Collaborator

@CoryCharlton I'm sorry but I cannot download the trace. It times out for me. Could you please just attach to the GH issue?

I wonder if adding the --no-compress argument helps?

@CoryCharlton
Copy link
Author

CoryCharlton commented Nov 10, 2022

@dobairoland Here's the trace:

trace.txt

Here's the output and trace from replacing -z with --no-compress (the result was the same failure):

output_no-compress.txt
trace_no-compress.txt

@pauldupuyjr
Copy link

I am also seeing this error with an Unexpected Maker ProS3 board using PlatformIO 2.5.5 (and platform_packages = tool-esptoolpy@~1.40300.0) when executing the "Upload Filesystem Image" task:

A fatal error occurred: Invalid head of packet (0x6F): Possible serial noise or corruption.
*** [uploadfs] Error 2

In Arduino IDE 1.8.19 using arduino-esp32fs-plugin 2.0.7 the result is:

->Writing at 0x00910000... (100 %)
->
->A fatal error occurred: Invalid head of packet (0x6F): Possible serial noise or corruption.
SPIFFS Upload failed!

However, in PlatformIO, when I "Erase Flash", add "board_build.partitions = default.csv" to platformio.ini configuration, and execute "Upload Filesystem Image" it seems to succeed:

Building in release mode
Building FS image from 'data' directory to build\esp32s3\spiffs.bin
/upload-test.txt
Looking for upload port...
Using manually specified: COM15
Forcing reset using 1200bps open/close on port COM15
Waiting for the new upload port...
Uploading build\esp32s3\spiffs.bin
esptool.py v4.3
Serial port COM15
Connecting...
Chip is ESP32-S3 (revision v0.1)
Features: WiFi, BLE
Crystal is 40MHz
MAC: f4:12:fa:42:0c:20
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x00290000 to 0x003fffff...
Compressed 1507328 bytes to 2855...
Writing at 0x00290000... (100 %)
Wrote 1507328 bytes (2855 compressed) at 0x00290000 in 11.4 seconds (effective 1061.8 kbit/s)...
Hash of data verified.

@CoryCharlton
Copy link
Author

Interesting, using default.csv did seem to work although it doesn't use the entire 16MB flash so I don't have enough room for my files.

Here's the partition layouts I was using:

default_16MB.csv * My board uses this by default
large_spiffs_16MB.csv
no_ota.csv

And for comparison the default.csv that works:

default.csv

@CoryCharlton
Copy link
Author

CoryCharlton commented Nov 12, 2022

Looks like the issue is possibly related to the default 16MB partition tables. I used the following custom partition table and it worked fine:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     36K,     20K,
factory,  app,  factory, 64K,     4M,
spiffs,   data, spiffs,  ,        10M,

Looks like this is actually an issue for https://github.com/espressif/arduino-esp32 (or is it?)

@pauldupuyjr
Copy link

Thanks @CoryCharlton for providing a working partition table example and for helping to solve this problem! Cheers,

@CoryCharlton
Copy link
Author

So this might still be an issue on the esptool side (based on this comment).

The specific issue appears to be related to the partition table completely filling up the 16MB flash. For instance I broke the working partition table mentioned here by extending the spiffs partition to ~12MB in order to completely use all available space.

# Name,   Type, SubType,  Offset,   Size,  Flags
nvs, data, nvs, 0x9000, 0x5000,
factory, app, factory, 0x10000, 0x400000,
spiffs, data, spiffs, 0x410000, 0xBF0000,

But I can make it work again by shrinking the spiffs partition by 1KB:

# Name,   Type, SubType,  Offset,   Size,  Flags
nvs, data, nvs, 0x9000, 0x5000,
factory, app, factory, 0x10000, 0x400000,
spiffs, data, spiffs, 0x410000, 0xBEFC00,

@dobairoland
Copy link
Collaborator

@CoryCharlton @SuGlider I don't think this is an issue with esptool nor Arduino.

If you take a look at the trace then you can see at the end that some serial data left in the buffer:

    70695f666c617368 2e6320323730200d | pi_flash.c 270 .
    0a                                | .

Invalid head of packet (0x6F) - 0x6f is ASCII for o. So actually serial communication is interrupted by an assertion from ROM (opi_flash.c, I guess 270 is the line number).

Honestly, I don't know what esptool could do about this. There is not enough information to print something more useful.

I'll try to ask around.....

@dobairoland
Copy link
Collaborator

It was pointed out to me that this is a duplicate of #745.

@dobairoland
Copy link
Collaborator

I'm able to write 16kB to the end of the flash region on an ESP32-S3 with 8MB flash as opposed to 16MB reported here.

`--> python -m esptool write_flash --no-compress 0x7fc000 ~/Work/zeroes.bin
esptool.py v4.3
Found 2 serial ports
Serial port /dev/ttyUSB0
Connecting...
Detecting chip type... ESP32-S3
Chip is ESP32-S3 (revision v0.0)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 7c:df:a1:e0:11:78
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x007fc000 to 0x007fffff...
Wrote 16384 bytes at 0x007fc000 in 1.6 seconds (84.3 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
.-(~/esp/esp-idf)----------------------------------------------------------------------------------------------------------------------------------------------------------(dragon@RolandPC)-
`--> python -m esptool write_flash --compress 0x7fc000 ~/Work/zeroes.bin
esptool.py v4.3
Found 2 serial ports
Serial port /dev/ttyUSB0
Connecting....
Detecting chip type... ESP32-S3
Chip is ESP32-S3 (revision v0.0)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 7c:df:a1:e0:11:78
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x007fc000 to 0x007fffff...
Compressed 16384 bytes to 39...
Wrote 16384 bytes (39 compressed) at 0x007fc000 in 0.2 seconds (effective 630.0 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

@skieast
Copy link

skieast commented Nov 21, 2022

I looked at this a bit and I think the issue is in the flasher_stub code. The routines that read_flash and write_flash change the spi method when above the first 16M flash. I think that this is happening 1 byte early. Meaning in the first 16M of flash. When I change the lines of code that check I am able to read and write to then end.
For example, commented out line is original.

    #if defined(ESP32S3)
      if (addr + len-1 > 0x00ffffff)
      // if (addr + len > 0x00ffffff)
        res = SPIRead4B(1, SPI_FLASH_FASTRD_MODE, addr, buf, n);
      else
        res = SPIRead(addr, (uint32_t *)buf, n);
    #else
      res = SPIRead(addr, (uint32_t *)buf, n);
    #endif // ESP32S3

@Evan5935
Copy link

So this might still be an issue on the esptool side (based on this comment).

The specific issue appears to be related to the partition table completely filling up the 16MB flash. For instance I broke the working partition table mentioned here by extending the spiffs partition to ~12MB in order to completely use all available space.

# Name,   Type, SubType,  Offset,   Size,  Flags
nvs, data, nvs, 0x9000, 0x5000,
factory, app, factory, 0x10000, 0x400000,
spiffs, data, spiffs, 0x410000, 0xBF0000,

But I can make it work again by shrinking the spiffs partition by 1KB:

# Name,   Type, SubType,  Offset,   Size,  Flags
nvs, data, nvs, 0x9000, 0x5000,
factory, app, factory, 0x10000, 0x400000,
spiffs, data, spiffs, 0x410000, 0xBEFC00,

can I get you to make me a partition table like the one you found worked for you? i have no idea what I am doing, but just need a 2mb app 12 mb fatfs or spiffs or whatever so I can get a project running on my ProS3 by UM

Thank You!

@CoryCharlton
Copy link
Author

can I get you to make me a partition table like the one you found worked for you? i have no idea what I am doing, but just need a 2mb app 12 mb fatfs or spiffs or whatever so I can get a project running on my ProS3 by UM

Thank You!

Here's a partition table with room for a 4MB app and a ~12MB SPIFFS partition: partitions.csv

Place the partitions.csv in your project directory next to the platformio.ini file. Edit your platformio.ini to add the following line:

board_build.partitions = partitions.csv

Complete platformio.ini example from one of my projects:

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[env]
platform = espressif32
board = um_feathers3
board_build.partitions = partitions.csv
framework = arduino
monitor_port = COM14
upload_port = COM14
lib_deps = earlephilhower/ESP8266Audio@^1.9.7

[env:debug]
build_type = debug
build_flags = -DARDUINO_USB_MODE=1 -DCORE_DEBUG_LEVEL=5

Then use the PlatformIO commands to perform the following steps: Erase Flase, Build Filesystem Image, Upload Filesystem Image

I'm on the Unexpected Maker Discord as Roland Deschain if you run in to any issues.

@beckmx
Copy link

beckmx commented Dec 2, 2022

@skieast hello there, I looked for those lines and I found them with .c extension, is that correct? I mean I changed those values but I got the same response, should I recompile esptool?

@skieast
Copy link

skieast commented Dec 2, 2022

You will need to recompile the flasher_stub routines that are in the flasher_stub folder. That is C code, the instructions are in the folder. Consists of downloading the ESP8266 compiler and sdk and the ESP32 idf. I don't think its really worth it at this time because you are only limited to not being able to read/write the last byte of 16M flash.

From README.md

# To Build

If you want to build the stub to test modifications or updates, here's how:

* You will need an ESP8266 gcc toolchain (xtensa-lx106-elf-) and toolchains for ESP32 and later chips (xtensa-esp32-elf-, riscv32-esp-elf-) on your PATH.

* Set the environment variables SDK_PATH to the path to an ESP8266 IoT NON-OS SDK directory (last stub was built with SDK v1.5.1).

* Set the environment variable IDF_PATH to the path to an ESP-IDF directory.

* Set any other environment variables you'd like to override in the Makefile.

* To build type `make`

Activating an ESP-IDF environment takes care of most of these steps (only the ESP8266 gcc toolchain has to be manually added to PATH).

# To Test

To test the built stub, you can run `make embed`, which will update the stubs in `esptool.py` to the newly compiled ones. Or there are some convenience wrappers to make testing quicker to iterate on:

* Running `esptool_test_stub.py` is the same as running `esptool.py`, only it uses the just-compiled stubs from the build directory.

* Running `run_tests_with_stub.py` is the same as running `test/test_esptool.py`, only it uses the just-compiled stubs from the build directory.

@beckmx
Copy link

beckmx commented Dec 2, 2022

Well I couldnt get it to compile but!! @CoryCharlton sample partition definately worked for me

Jason2866 added a commit to Jason2866/esptool that referenced this issue Dec 3, 2022
@Jason2866
Copy link
Contributor

@beckmx You can try my fork https://github.com/Jason2866/esptool
I did the change and let github actions build the stub.

nhjschulz pushed a commit to nhjschulz/esp-rgb-led-matrix that referenced this issue Dec 4, 2022
* Enabled PSRAM memory (+8MB usable RAM)
 - This setting is crucial:
     board_build.arduino.memory_type = qio_opi
* Use 16MB flash partition scheme
 - A custom one with 16MB-1Byte size is used to workaround a tool
   bug when trying to upload full 16MB
   see espressif/esptool#795 (comment)
- Add PSRAM to info web page
@henrygab
Copy link

I don't think its really worth it at this time because you are only limited to not being able to read/write the last byte of 16M flash.

I do hope this gets reviewed, validate, and merged ... because I have a few ESP32-S3 with 32MB of flash, and I've had to artificially limit use of those to <16MB.

espressif-bot pushed a commit that referenced this issue Dec 28, 2022
SPIWrite4B and SPIRead4B functions are required for flash sizes bigger
than 16MB. This fix corrects the boundaries so SPIWrite and SPIRead
would be used for the whole 16MB area.

The octal flash support for 32 MB chips
(#795) will be added in a
follow-up commit.

Closes #745
Jason2866 added a commit to Jason2866/esptool that referenced this issue Dec 28, 2022
* fix(setup): Use latest reedsolo package which can be installed with Python3.10 and Cython

Closes espressif#711

* feat(ci): Publish development releases with custom pipeline

* fix(ci): The development release job should not run by default

Gitlab rules:if adds the job if any of the rules are true.

* fix(ci): Merge two "ci" directories and build_tools into one

* fix(secure download mode): Fix SDM detection on S2/S3

* fix(secure download mode): Reconnect if ROM refuses to respond

Closes espressif#813

* fix(flasher_stub): Correct boundaries for SPIWrite4B and SPIRead4B

SPIWrite4B and SPIRead4B functions are required for flash sizes bigger
than 16MB. This fix corrects the boundaries so SPIWrite and SPIRead
would be used for the whole 16MB area.

The octal flash support for 32 MB chips
(espressif#795) will be added in a
follow-up commit.

Closes espressif#745

Co-authored-by: Roland Dobai <roland@espressif.com>
Co-authored-by: radim.karnis <radim.karnis@espressif.com>
Jason2866 added a commit to Jason2866/esptool that referenced this issue Dec 28, 2022
* Tasmota changes

* stubs updated

* S3 16MB fix (#4)

* fix(setup): Use latest reedsolo package which can be installed with Python3.10 and Cython

Closes espressif#711

* feat(ci): Publish development releases with custom pipeline

* fix(ci): The development release job should not run by default

Gitlab rules:if adds the job if any of the rules are true.

* fix(ci): Merge two "ci" directories and build_tools into one

* fix(secure download mode): Fix SDM detection on S2/S3

* fix(secure download mode): Reconnect if ROM refuses to respond

Closes espressif#813

* fix(flasher_stub): Correct boundaries for SPIWrite4B and SPIRead4B

SPIWrite4B and SPIRead4B functions are required for flash sizes bigger
than 16MB. This fix corrects the boundaries so SPIWrite and SPIRead
would be used for the whole 16MB area.

The octal flash support for 32 MB chips
(espressif#795) will be added in a
follow-up commit.

Closes espressif#745

Co-authored-by: Roland Dobai <roland@espressif.com>
Co-authored-by: radim.karnis <radim.karnis@espressif.com>

* Update stub_flasher_32s3beta2.json

* Update build_esptool.yml

* Update build_esptool.yml

* stubs updated

Co-authored-by: Github BUILD <github-actions@github.com>
Co-authored-by: Roland Dobai <roland@espressif.com>
Co-authored-by: radim.karnis <radim.karnis@espressif.com>
Jason2866 added a commit to tasmota/esptool that referenced this issue Dec 28, 2022
* S3 16MB fix (#4)

* fix(setup): Use latest reedsolo package which can be installed with Python3.10 and Cython

Closes espressif#711

* feat(ci): Publish development releases with custom pipeline

* fix(ci): The development release job should not run by default

Gitlab rules:if adds the job if any of the rules are true.

* fix(ci): Merge two "ci" directories and build_tools into one

* fix(secure download mode): Fix SDM detection on S2/S3

* fix(secure download mode): Reconnect if ROM refuses to respond

Closes espressif#813

* fix(flasher_stub): Correct boundaries for SPIWrite4B and SPIRead4B

SPIWrite4B and SPIRead4B functions are required for flash sizes bigger
than 16MB. This fix corrects the boundaries so SPIWrite and SPIRead
would be used for the whole 16MB area.

The octal flash support for 32 MB chips
(espressif#795) will be added in a
follow-up commit.

Closes espressif#745

Co-authored-by: Roland Dobai <roland@espressif.com>
Co-authored-by: radim.karnis <radim.karnis@espressif.com>

* Update stub_flasher_32s3beta2.json

* Update build_esptool.yml

* Update build_esptool.yml

* stubs updated

Co-authored-by: Github BUILD <github-actions@github.com>
Co-authored-by: Roland Dobai <roland@espressif.com>
Co-authored-by: radim.karnis <radim.karnis@espressif.com>

* stubs updated

* stubs updated

Co-authored-by: Github BUILD <github-actions@github.com>
Co-authored-by: Roland Dobai <roland@espressif.com>
Co-authored-by: radim.karnis <radim.karnis@espressif.com>
@CoryCharlton
Copy link
Author

Confirming that this commit resolves my issue with 16M flash.

I was going to close this issue but it looks like you're keeping it open for an issue with 32M flashes that is not the same as my original issue.

Feel free to close this if I am mistaken.

dobairoland added a commit to espressif/esptool-legacy-flasher-stub that referenced this issue May 30, 2024
SPIWrite4B and SPIRead4B functions are required for flash sizes bigger
than 16MB. This fix corrects the boundaries so SPIWrite and SPIRead
would be used for the whole 16MB area.

The octal flash support for 32 MB chips
(espressif/esptool#795) will be added in a
follow-up commit.

Closes espressif/esptool#745
dobairoland added a commit to espressif/esptool-legacy-flasher-stub that referenced this issue May 30, 2024
Octal flash support tested up to 32MB. Quad flash support is limited to
16MB at this moment.

Closes espressif/esptool#795

Closes espressif/esptool#745

The next release will solve espressif/esp-idf#10323 as well.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants