NodeMCU 3.0 spiffsimg not working?

[albarrett]

Please bear with me, I don’t know if this is an issue with NodeMCU or me! I have been in touch with @TerryE regarding this trying to understand the issue but have yet to find a solution.

I have been using my own Linux build environment for NodeMCU 2.2 for a while and the process would generate 3 files, 2 bin files and a spiffsimg file. I could then flash all 3 of these files to my ESP8266 using esptool and all the files I had put into local/fs folder would be available on the device.

I am now attempting to do the same thing with NodeMCU 3.0 but I am having 2 problems.

Firstly, I cannot get the ‘make’ process to generate the spiffsimg file. What do I need to do to get this to generate?

Secondly, if I create this file manually using the spiffsimg tool, when I flash it to the device as I did with 2.2 the device doesn’t see any files in its file structure. However, if I take this img file and flash it using bin files from a 2.2 build the files can be seen and used, which is why I am questioning if there is an issue with 3.0 or me?

I am aware there is the new partition tool that I can use, but if possible, I would like to stick to the same workflow I was using for 2.2 for several reason I won’t bore you with.

Any assistance on the above would be greatly appreciated.

Expected behavior

View and use text and HTML files from spiffsimg file I have generated and flashed onto the device.

Actual behavior

Flashing the img file is successful, however, no files can be seen or used. Device formats its own blank filesystem, as if the img files doesn’t exist, which I can upload files to.

Test code

N/A

NodeMCU version

Master branch (NodeMCU 3.0)

Hardware

ESP8266-12S dev board

[KT819GM]

Can confirm, on 043046d spiffsimg is not created when files available on $(TOP_DIR)/local/fs/*) while using make on Linux environment.

[TerryE]

Alex, the reason that I've put my name against this is that I'll fix this in my next push to dev.

[albarrett]

@KT819GM Thank you for taking the time to read and comment. I have pulled the commit you suggested from dev branch, left the user_config.h and user_modules.h files as default and just added some files to local/fs and local/lua and run make as suggested, however, unfortunately i do only get the 2 bin files and not the spiffsimg file. Am i correct in assuming that as the default user_config.h file has #define BUILD_SPIFFS uncommented that i am to expect the img file to be generated?

@TerryE Thank you for your comment. So am i to assume that the issue is not me? Any idea on the time frame for your next push to dev?

[momkey02]

I am now attempting to do the same thing with NodeMCU 3.0 but I am having 2 problems.

Firstly, I cannot get the ‘make’ process to generate the spiffsimg file. What do I need to do to get this to generate

The same question as you,

[TerryE]

@albarrett @KT819GM @momkey02 fixing this is stalled on #2960. More feedback needed, please.

[albarrett]

@TerryE, this issue randomly came up so i thought i would just check on progress, as this issue is still open i assume the issue is still ongoing? I spotted post #3066 about a new master release and was hoping this fix would be included in that release, do you know if thats going to be the case? Would be great to be able to finally use 3.0, im still using 2.2.1 whilst waiting for this to be resolved. Thanks.

[TerryE]

Alex, the reason that I've put my name against this is that I'll fix this in my next push to dev.

Ahhh, mea culpa, and almost 10 months since the issue was already logged. I will put this fix in next. Sorry guys. This also cuts across #3147, so I would prefer to sort out this issue of harmonising firmware and host spiffs within a single thread of discussion and one PR. The scope of this will be:

• The local/fs and local/lua target should work as with the SDK 2.x behaviour and the generated SPIFFS image should work with the firmware.
• I suggest that we drop the outdated spiffs image parameters.
• @pnkowl raised some SPIFFS issues in SPIFFs File Corruption #3096, and to be honest I don't have the bandwidth to track these down. I doubt that any of the other committers will step forward here, but at least there are some issues that I want to discuss below and I think that I discussed with @pjsg

[zxsoft]

@TerryE I've met this problem everytime I tried to build a nodemcu firmware. And I don't know another way to place the local/fs and local/lua files in the esp8266 mcu. So would you please tell me another way to manually do that? All I want to know is the way to place "init.lua" and other files into the mcu without using the "file" API. I just want to flash the esp8266 mcu with the node-firmware I built and upload all files I needed into that mcu using the same flash tool. So would you teach me that please? Thank you so much! It bothered me for such a while...

[nwf]

@zxsoft: What, specifically, is your issue? I just tried on dev (plus some hopefully irrelevant changes): make spiffs-image did what seems to be the right thing:

• it enumerated local/lua and cross-compiled a LFS image into local/fs/LFS.img
• it enumerated local/fs and constructed a SPIFFS image at tools/spiffsimg/spiffsimg

I'd expect to be able to pass tools/spiffsimg/spiffsimg through nodemcu-partiton.py --spiffs_file and have the right thing happen?

(ETA: I usually do this quite by hand rather than using the in-tree Makefiles because I have multiple projects that all cluster around a single firmware tree rather than multiple firmware trees each containing a mostly-lua project. I can show you how to set that up, too, but it's really "just do what the build system does but with slightly different flavor" and so if the build system isn't doing the right thing for you, there's probably a bigger problem.)

[zxsoft]

@nwf Thanks a lot for your reply!

I used a docker build environment, the image name was "marcelstoer/nodemcu-build", the node-firmware git commit hash on which I used was ”136e09739b835d6dcdf04034141d70ab755468c6“.

I cloned the nodemcu/nodemcu-firmware using the above commit hash on github, without editing any files. I then added two files, "local/fs/fs.lua" and "/local/lua/lua.lua", and write a “print('hello world')" line in each of them. Then, I run "make" command. It successfuly generated two files 0x00000.bin and 0x10000.bin in the bin directory. I flashed the image into esp8266, the nodemcu-firmware worked very well but I didn't find the two files I added.

I've just tried your suggested command "make spiffs-image”, but seems something went wrong:

root@nodemcu:~/nodemcu-firmware# make spiffs-image
make: Entering directory '/tmp/nodemcu-firmware'
make[1]: Entering directory '/tmp/nodemcu-firmware/tools'
make[2]: Entering directory '/tmp/nodemcu-firmware/tools/spiffsimg'
HOSTCC /tmp/nodemcu-firmware/tools/spiffsimg/main.c
make[2]: Leaving directory '/tmp/nodemcu-firmware/tools/spiffsimg'
Unable to calculate where to put the flash image, and % present in filename: fatal error
Makefile:74: recipe for target 'spiffsscript' failed
make[1]: *** [spiffsscript] Error 1
make[1]: Leaving directory '/tmp/nodemcu-firmware/tools'
Makefile:398: recipe for target 'spiffs-image' failed
make: *** [spiffs-image] Error 2
make: Leaving directory '/tmp/nodemcu-firmware'
root@nodemcu:~/nodemcu-firmware#

Then I checked the directory you mentioned, and found that there was indeed a LFS.img file in the local/fs directory (Which didn't appear when I just use the make command, or using the "build" script provided by the docker image), so does that mean I can ignore the error messages above?

There was also the file "tools/spiffsimg/spiffsimg", but it's an executable program file, not an SPIFFS image.

I wanted to list the files in the LFS.img to check wether it's a good image.So I used the command:

/tmp/nodemcu-firmware/tools/spiffsimg/spiffsimg -f LFS.img -l

and got an error:

file size not multiple of erase block size: fatal error

So I dought the file was not generated successfully. The size of LFS.img file is 279Byte.

I don't know how to deal with the LFS.img file. Should I flash the LFS.img manually into the esp8266 mcu?

When I use the "build" script the docker image provided, there will be four files in the bin directory.

0x00000.bin 0x10000.bin nodemcu_float_release_20210228-1437.bin nodemcu_float_release_20210228-1437.map

The code to build the extra two files in the "build" script is:

# build
make WRAPCC="$(which ccache)" $BUILD_MAKE_TARGETS
cd bin
srec_cat -output nodemcu_${BUILD_TYPE}"${IMAGE_NAME}".bin -binary 0x00000.bin -binary -fill 0xff 0x00000 0x10000 0x10000.bin -binary -offset 0x10000
# copy and rename the mapfile to bin/
cp ../app/mapfile nodemcu${BUILD_TYPE}_"${IMAGE_NAME}".map
cd ../

What I really wanted is to pre-build "local/fs" files into the all-in-one image "nodemcu_float_release_20210228-1437.bin", especially the "init.lua" file. So I can flash the image and get the esp8266 mcu ready to run as soon as it's been powered up.

I worked hard on this for a while but not managed to get it done. So I'm very looking forward to your help. Thank you so much!

[zxsoft]

@nwf I read you "ETA" section and very interested in the way you manage the firmware tree. So how are you doing that? If I want to flash the file system in to esp8266 mcu using download tool (not using Esplorer IDE or the file API to write init.lua file in it), what's the best practise?

[nwf]

https://github.com/cmukgb/ctfws-timer-iot/blob/master/README-flashing.rst is the closest thing I have to a writeup of the process I use; there may some rough spots.

And yes, sorry, you're absolutely right that tools/spiffsimg/spiffsimg is the imaging tool itself; I apologize for the error. The filesystem images should be in bin/ and be named something like 0xc0000-32mb.img; the exact name will depend on the size of the firmware image constructed.

In principle an all-in-one file should be constructable; presumably, it looks like "concatenate the firmware image, some padding, the filesystem image, some more padding, and then the esp-init data" but I suspect you'll be better off (it'll be a little faster) using the multiple files and the esptool and nodemcu-partition tools to flash in stages.

Hope that helps!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gxghg]

@zxsoft 你的好了吗？你如何解决的？我这边也需要将lua代码写入bin中，3.0貌似有问题