mkspiffs crashes when invoked with negative size parameter

[maxgerhardt]

As detailed in https://community.platformio.org/t/building-uploading-spiffs-fails-error-3/11160/6?u=maxgerhardt, for the platformio.ini:

[env:d1_mini_pro]
platform = espressif8266
board = d1_mini_pro
framework = arduino

and creating a data/ folder with one sample file inside it, the pio run -t uploadfs task will fail with a fatal error when creating the 14MB SPIFFS section, as dictated by the linker file,

"mkspiffs" -c data -p 256 -b 8192 -s -2121728 .pio\build\d1_mini_pro\spiffs.bin
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Because it cannot allocate -2121728 bytes. The error lies within the correction code for the SPIFFS_START and SPIFFS_END variables which is done in this code

platform-espressif8266/builder/main.py

Lines 88 to 111 in b866c29

	def fetch_spiffs_size(env):
	ldsizes = _parse_ld_sizes(env.GetActualLDScript())
	for key in ldsizes:
	if key.startswith("spiffs_"):
	env[key.upper()] = ldsizes[key]
	assert all([
	k in env
	for k in ["SPIFFS_START", "SPIFFS_END", "SPIFFS_PAGE", "SPIFFS_BLOCK"]
	])
	# esptool flash starts from 0
	for k in ("SPIFFS_START", "SPIFFS_END"):
	_value = 0
	if env[k] < 0x40300000:
	_value = env[k] & 0xFFFFF
	elif env[k] < 0x411FB000:
	_value = env[k] & 0xFFFFFF
	_value -= 0x200000 # correction
	else:
	_value = env[k] & 0xFFFFFF
	_value += 0xE00000 # correction
	env[k] = _value

The code transforms the original values

k = SPIFFS_START, value = 0x40400000
k = SPIFFS_END, value = 0x411fa000

to

SPIFFS_END is -0x6000
SPIFFS_START is 0x200000
Diff is -0x206000

thus yielding a negative length for the SPIFFS section. This error seems to occur for large 16MB / 128MBit flash chips. There might be other cases in which this correction logic is wrong?

When esptool flash starts at 0 as the comment says, why not subtract the start address of the flash, 0x40200000, from both addresses? I don't understand the correction logic at the moment.

[stuartpittaway]

Confirm that I also get this bug with d1 mini pro - 128mbit flash

Pretending the mini pro only has 64mbit of flash works
board_build.ldscript = eagle.flash.8m7m.ld

[maxgerhardt]

Still occuring per https://community.platformio.org/t/fail-to-create-littlefs-on-d1-mini-pro-board/16539 CC @valeros

[FIAV1]

Same problem here with 16MB flash chips. Any news on the fix?

[zgjsntsm]

When using littlefs, build filesystem image will not prompt an error, but the bin file is wrong, file size is 4GB

[maxgerhardt]

CC @valeros in the linked thread people are still experiencing this issue

[infrafast]

CC @valeros in the linked thread people are still experiencing this issue

I especially observed that when you make the filesystem with the workaround, if you try to do an httpupdate or an update by opening locally a file that was stored in the filesystem, it fails.

example of code via localfile:

File file = LittleFS.open("firmware.bin", "r");
while (file.available()) {
uint8_t ibuffer[128];
file.read((uint8_t *)ibuffer, 128);
Update.write(ibuffer, sizeof(ibuffer));
DEBUG_PRINT("\nwritting");
}
if (Update.end(true)) {
DEBUG_PRINT("\nupdate.end returned false");
}else{
DEBUG_PRINT("\nupdate.end returned false");
}

output:
writting
writting
...
writting
update.end returned false

example of code via httpupdate:

} if (upload.status == UPLOAD_FILE_WRITE) {
//UPDATE WRITE FIRMWARE
if (Update.write(upload.buf, upload.currentSize) != upload.currentSize) {
DEBUG_PRINT("\nFW write error");
}
}

output:
FW write error

[ctbenergy]

I changed the bitmask from 0xFFFFFF to 0x1FFFFFF in %USERPROFILE%\.platformio\platforms\espressif8266\builder\main.py

     elif env[k] < 0x411FB000: 
         _value = env[k] & 0x1FFFFFF 

[landorg]

Hi.
Any news on this?
Unfortunately none of the above mentioned workarounds work for me. Thus I'm unable to work with pio & my d1_mini_pro's.

@ctbenergy 's workaround gives me an md5 mismatch and corrupted program.
Usnig eagle.flash.8m7m.ld as @stuartpittaway gives me the same result.

I also tried littlefs without luck.

If anyone could point out how I'd calculate the values for mkspiffs / esptool. That would be very much appreciated.
Thanks

Edit:
What works: eagle.flash.4m1m.ld

[Riffer]

The bug still exists. Interestingly the upload with Arduino IDE (the old one) works. So, what's the problem to fix this here?

[maxgerhardt]

^ -- CC @valeros

[Riffer]

The bitwise shifting does not work and I would like to use the full capacity of the device.
Has anyone a handy patch or a simple solution instead of low-level bit operations? (I absolutely understand this can speed up a process but... seriously in such a script? Seems someone wanted to impress and overshooted.)

[dzsoni]

I changed the bitmask from 0xFFFFFF to 0x1FFFFFF in %USERPROFILE%.platformio\platforms\espressif8266\builder\main.py

     elif env[k] < 0x411FB000: 
         _value = env[k] & 0x1FFFFFF 

It's worked for me.

[TronTronix]

Mich hatt es auch erwischt mit mehreren 16 MB Flash-Chips von Aliexpress ("D1 MINI PRO 16M")
platformio.ini

[env:d1_mini_pro]
platform = espressif8266
board = d1_mini_pro
board_build.filesystem = littlefs
board_build.ldscript = eagle.flash.16m14m.ld
framework = arduino

Die obere Änderung mit 0x1FFFFFF geht bei mir nicht.
Da ich auf der schnelle nicht erkennen kann ob diese Funktion für andere Sachen gebraucht wird bin ich von abgerückt da Veränderungen zu machen.

%USERPROFILE%\.platformio\platforms\espressif8266\builder\main.py 103-115

    # esptool flash starts from 0
    for k in ("FS_START", "FS_END"):
        _value = 0
        if env[k] < 0x40300000:
            _value = env[k] & 0xFFFFF
        elif env[k] < 0x411FB000:
            _value = env[k] & 0xFFFFFF
            _value -= 0x200000  # correction
        else:
            _value = env[k] & 0xFFFFFF
            _value += 0xE00000  # correction

        env[k] = _value

Und Fange einfach den Fehler ab und ändere ihn in ein richtigen Wert zur Weiterverarbeitung, ist zwar nicht das optimalste aber es funktioniert.

    # esptool flash starts from 0
    for k in ("FS_START", "FS_END"):
        # Tron
        print("k = %s, value = %s" % (k, hex(env[k])))

        _value = 0
        if env[k] < 0x40300000:
            _value = env[k] & 0xFFFFF
        elif env[k] < 0x411FB000:
            _value = env[k] & 0xFFFFFF
            _value -= 0x200000  # correction
        else:
            _value = env[k] & 0xFFFFFF
            _value += 0xE00000  # correction

        # Tron
        if _value == -0x6000:
            _value = env[k] - 0x40200000
            print("neu = %s, value = %s" % (k, hex(_value)))

        env[k] = _value

Es muß ja ein Grund geben das man es nicht ändert seit Jahren, aber so wäre es sicherlich vertretbar für alle. („Das sind nur Zwei Zeilen ohne die Anzeige“)

        if _value == -0x6000:
            _value = env[k] - 0x40200000

main.zip

[innir]

This issue is not SPIFFS specific, it happens with LittleFS as well and still exists, see https://github.com/platformio/platform-espressif8266/blob/master/builder/main.py#L92-L115

@valeros any chance to get that fixed soon?

[bphermansson]

Confirm that I also get this bug with d1 mini pro - 128mbit flash

Pretending the mini pro only has 64mbit of flash works board_build.ldscript = eagle.flash.8m7m.ld

This works for me.

[iUltimateLP]

Adding to the discussion that eagle.flash.4m1m.ld works for me. 4mb of flash are enough for my cases, even though it's not optimal, hoping to see a fix sometime. The 8mb linker script produces MD5 errors and the 16mb linker script produces the original issue in this thread.