A Lua Cross-Compile Web Service Generates Unusable Images

[pnkowl]

Expected behavior

node.flashindex("_init")() -- do it from flash results in access to LFS

Actual behavior

node.flashindex("_init")() -- do it from flash

Communications Error - Check baud rate
 ets Jan  8 2013,rst cause:4, boot mode:(3,6)

Test code

-- add code here
-- create zip file with stock _init.lua 
-- create LFS image using https://blog.ellisons.org.uk/article/nodemcu/a-lua-cross-compile-web-service/
-- upload LFS.img file to esp8266
-- flash LFS image
=node.flashreload("LFS.img")
-- initialize
dofile("_init.lua")  -- local
node.flashindex("_init")()   -- do it from flash
--Communications Error - Check baud rate
-- ets Jan  8 2013,rst cause:4, boot mode:(3,6)
--

NodeMCU startup banner

NodeMCU 3.0.0.0 built on nodemcu-build.com provided by frightanic.com
	branch: master
	commit: 71a182caa7841cbb478ed90ede526dc881943c80
	release: 
	release DTS: 202001061321
	SSL: false
	build type: float
	LFS: 0x20000
	modules: adc,bit,cron,dht,file,gpio,http,mqtt,net,node,ow,pwm2,rtcmem,rtctime,sntp,struct,tmr,uart,wifi
 build 2020-02-20 03:19 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

Hardware

ESP8266 Wemos D1 mini (ESP-12F) plug and play, just connect to usb adapter

• LFS200428_bug1a.zip
• LFS200428_bug1a (img).zip

When

Issue did not exist on April 26. Images created prior to today still work fine. Using the web service on zip files that created working images in the past, no longer create working images now

[pnkowl]

Thanks for all the work on NodeMCU

[TerryE]

using the web service on zip files that created working images in the past, no longer create working images now.

Sorry, but the cross-compile service is something that I created outside of the project which is why it is hosted on a personal RPi4 on my (ellisons) domain rather than a project one. A limitation or feature if you will is that it supports the current dev and master versions and in fact an automated remake is triggered whenever a PR is merged into ether of these. (I will check the logs to make sure that the last make worked properly).

This can mean that any non-backwards compatible change can mean that the generated LFS images can fail to load with a firmware build on an older version of dev and master.

Off the top of my head the only change that I have added to luac.cross is the extra check to throw a compile error if duplicated module names occur in a file list.

Really the best thing to do is to generate your LFS images locally on your own PC. The documentation explains how you can use a range of compile options to build this locally using WSL, MinGW, Cygwin or native MSC.

My own challenge is that I don't really own any WinX H/W. I have kept one old Laptop which can dual boot into Win7 and which I've used to checked these out. I have asked for anyone else who does use WinX as their primary development environment to take this on, but no takers :(

Needs a bit more thought.

[pnkowl]

Thanks for getting to this.

First things first. I have extra hardware at home. I am most comfortable in windows. I have 2 RPi-2bs and 2 PiZeroWs. Not sure if they have enough horsepower. I periodically considered running the cross compiler on a windows box but without a bit of guidance, I was apprehensive. Going from hello world to the more advanced stuff takes a bit of trial and error for regular people. I also believe that your service is a big help to users. Supporting that with feedback is a honorable thing for me to do. I would also consider getting a domain and setting up a secondary/backup service in case 1 is down...

Second thing. I generated the sample LFS again. Here is a symptom I did not include originally. I expected a list of "module" names and get this instead with newly compiled images. And of course the esp panics if I try to index to what I expect to be there.
`dofile("_init.lua")

for k,v in pairs(LFS._list) do print(k,v) end
1 userdata: 0x4027c4e8
2 userdata: 0x4027c1b4
3 userdata: 0x4027c3ac
`

[pnkowl]

One last thing, brand new firmware also

NodeMCU 3.0.0.0 built on nodemcu-build.com provided by frightanic.com
	branch: master
	commit: 3d917850180f67adc7f2c6b5d00f27c152e7194c
	release: 
	release DTS: 202002231746
	SSL: false
	build type: float
	LFS: 0x40000
	modules: adc,bit,cron,dht,enduser_setup,file,gpio,mqtt,net,node,ow,pwm2,rtcmem,rtctime,sntp,struct,tmr,uart,wifi
 build 2020-05-01 13:41 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

[pnkowl]

One additional thought. Although Marcel (https://nodemcu-build.com/index.php) updated the front end to support 256k LFS images (nice) in the last day (so I would hope the back end is current), is it possible that firmware build service and the LFS web service are misaligned regarding "the current dev and master versions"? Custom firmware build work with 4/26 LFS images (and earlier), just not the new ones (starting 4/28 for me).

[nwf]

@pnkowl: The resources required by luac.cross are so small that, barring some engineering challenges, they'd almost fit on the ESP. That is, your Pi, of any generation, is more than sufficiently powerful.

[jiri-koptik]

@TerryE : Having the same problem as @pnkowl, even using latest firmware build (2020-05-06 07:43) from Marcel (https://nodemcu-build.com/index.php) and creating the LFS via TerryE web-service. "Old" LFS builds from April 2nd work fine with this firmware build.

• LFS._list returns only userdata:"some address" instead of the name of file
• running "node.flashindex'_init'()" results in reboot

Thanks for supporting us!

[TerryE]

There's definitely something broken which I am retracking down. Sorry for the delay

[pnkowl]

How is the solution progressing?

Thoughts

1. It may be helpful for users to know that the dev branch works. I did not know this until yesterday when I was checking out softuart (only available in dev) and tried the service on a whim. What is the risk of using Dev?

2. It may be helpful to post an ETA on the website (best guess only). Hopefully everyone understands that stuff happens and demands on developer time can change without notice)

3. Perhaps it is time to update the website to warn folks of this issue and explain the possible workarounds (e.g. dev branch, which I believe requires dev branch firmware). Without knowing about this, my development efforts have been directed elsewhere for the last 2 weeks.

4. I am open to hosting a secondary location for this service. I do understand that I would need to get a domain and ensure that my local network security is up to snuff (open incoming port exposure). I believe I have all the hardware required, but would need a good bit of mentoring to get it up and running. Or to even understand the real scope of what I am offering.

5. Lastly, I recall having read comments that the service is not used that much. For the longest time, given the brevity of the setup and usage instructions, I did not pursue LFS. I was fearful of a steep learning curve. I believe my experience could have been smoother. Given the fact that I have so little experience with all of this, it may be beneficial for me to take a crack at "LFS for dummies". But guess what? I am in the same newbie state for adding help to nodemcu.readthedocs. The website has a style, that seems to me, leaves bits of the stuff newbies stumble over unsaid. Balancing the needs of the utterly clueless to advanced developers who simply need reference material is a daunting and never ending task. I get that. Are there best practices for (1) adding a new page and (2) attaching it to an existing page to ensure that the initial PR is reasonably close to the expectations of those who "know best"? Can the necessary learning be done in minutes or an hour rather than days to weeks for a "simple" change to the docs?

Notwithstanding the comments above, thanks to all for spending thousands of hours and making the hard choices on what gets your time in any moment.

[TerryE]

@pnkowl I am just finishing off #3078. This one is next.

[pnkowl]

FYI.

LFS Dev branch is not dead like master, but not as expected either.
I tried 3 versions of telnet and none worked.
All returned a never ending stream of the following to the putty termial:
"Pipe: 0x3fff2f38Pipe: 0x3fff2f38Pipe: 0x3fff2f38Pipe: 0x3fff2f38Pipe: 0x3fff2f38Pipe: "

Note that Firmware Dev Branch with "simple" telnet (run without LFS) seems ok. BTW, FTP seems ok....

Moved to #3108

[TerryE]

@pnkowl, thanks for all of this great info. I've got all of the error handling stuff working now, so I will get onto this next 😄

[TerryE]

@pnkowl @jiri-koptik, can you double check the service now. I've just done some load tests against a current cloud builder dev build and this works fine.

[pnkowl]

@TerryE, okay, will do. I was wondering how the notice of the change would ripple through the various service providers. Thanks for the ping.

[pnkowl]

@TerryE, You reference the dev branch in the "give it a try" announcement above. This issue was opened against the master branch. It is my understanding that the dev branch was never afflicted with the same issue (irrepective of some of my comments above). Should master branch be working also and therefore be tested as well? If not, is there an estimate for when the master branch will be re-supported through the web service? I am just trying to be clear on intent and expectations. Also, does this also fix the ftp/telnet issue #3108? I originally thought that was the purpose, but now I am entirely unsure. That all being said, I will test both dev/dev and master/master configurations. Thanks for your time on this.

[TerryE]

Should master branch be working also

@pnkowl, we are just trying to clear down any outstanding issues impacting dev that are tagged "next release". We will then cut dev to master, and this will clear down any master issues.

[pnkowl]

@pnkowl @jiri-koptik, can you double check the service now. I've just done some load tests against a current cloud builder dev build and this works fine.

Is presumably a request to test the firmware/LFS master/master case, which is this issue.

@pnkowl, we are just trying to clear down any outstanding issues impacting dev that are tagged "next release". We will then cut dev to master, and this will clear down any master issues.

This implies a dev/dev test case.

But dev/dev was not broken for this issue.

So I am at a loss as to what testing is being requested at this time regarding this issue. Is there a suspicion that master/master may work and should be tested?

[TerryE]

OK, that explains why it works fine. 😄

[TerryE]

I had a problem on the RPi4 server which runs etckeeper but one of my regular apps dumps big updates to its /etc subfolder rather than its /var/lib one, and the git logs filled my root partition. Bugger. All fixed now.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.