Dev Branch Telnet, FTP Server Compatibility

[pnkowl]

Expected behavior

Dev branch to support Telnet, FTP and Web server enabled at the same time. Master branch does.

Actual behavior

Dev branch panics when FTP filelist requested while Telnet module is "open"

ftpserver.lua obtained from here
https://github.com/nodemcu/nodemcu-firmware/blob/dev/lua_modules/ftp/ftpserver.lua
telnet_pipe.lua obtained from here
https://github.com/nodemcu/nodemcu-firmware/tree/dev/lua_examples/telnet

Test code

-- add code here
2 second delay to stop code, use s()
> s()
> print(uart.setup(0, 115200, 8, 0, 1, 1 ))
115200
> station_cfg={}; station_cfg[9]={ssid="myssid",pwd="mypass",save=false}; wifi.setmode(wifi.STATION); wifi.sta.config(station_cfg[9]); wifi.sta.connect()
> print(wifi.sta.getip())
192.168.1.17	255.255.255.0	192.168.1.1
> node.flashindex("telnet_pipe")().open()
> Telnet server started (41888 mem free, 192.168.1.17)
print("open telnet connection with putty")
open telnet connection with putty
> node.flashindex("ftpserver")().createServer("test","12345") 
print("FTP file list request from Filezilla")
PANIC: unprotected error in call to Lua API (/tmp/luac_dirCaARcZ/ftpserver.lua:120: attempt to index a function value)

ftp:120 is
            socket:on("receive", function(socketObj, dataObj)

other occasions yield
> PANIC: unprotected error in call to Lua API (/tmp/luac_dirCaARcZ/ftpserver.lua:484: attempt to index a function value)
ftp:484 is
      skt:send(rec)

web ide fails with lines referencing socket also (not included here)

Telnet can be run as local Lua code with the same effect

FTP is too big to run from local Lua code.

NodeMCU startup banner

NodeMCU 3.0.0.0 built on nodemcu-build.com provided by frightanic.com
	branch: dev
	commit: 5e2ea5a226ae30b066968efd27e4e2510e6cd6f4
	release: 
	release DTS: 202005120049
	SSL: false
	build type: float
	LFS: 0x40000 bytes total capacity
	modules: adc,bit,cron,dht,file,gpio,mqtt,net,node,ow,pwm,rtcmem,rtctime,sntp,softuart,struct,tmr,uart,wifi
 build 2020-05-12 14:43 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

Dev branch LFS from cross compiler web service

Hardware

Wemos D1 Mini (esp8266-12F)

Software (examples/modules)

Will not be line for line matches with dev branch sources, but are the same code
telnet_pipe.lua
ftpserver.lua
200519a.zip

[TerryE]

Sorry, I've just deleted my previous reply, as I misread the issue and responded incorrectly. My bad, I will look into this. But I am getting a bit tired from all of the hours that I am putting in at the mo. Maybe it's time to take a few days break and read a book or two.

[TerryE]

I have just built and downloaded an image including _init, ftp and telnet and after reflashing:

> node.flashindex'_init'() -- setup so Lua LFS methods work and require also resolves from LFS
> =node.heap()
41512
> require'telnet_pipe':open('xxx','yyy',2323)
> ......Telnet server started (41216 mem free, 192.168.1.200)
require'ftpserver'.createServer('terry','terry',true)
-- now do everything else from the telnet session, until ...
=node.heap()
40072
> -- everything has been GCed except the growth in reserved Lua registry slots

And the telnet session (note that the FTP server diagnostics are coming out on the stdout telnet session):

$ telnet 192.168.1.200 2323
Trying 192.168.1.200...
Connected to 192.168.1.200.
Escape character is '^]'.
Welcome to NodeMCU world (40136 mem free, 192.168.1.200)

> 25640	Server created: (userdata) userdata: 0x3fff22c0
>
> for k,v in pairs(package.loaded) do print(k,v) end
package	table: 0x3fff0de8
telnet_pipe	table: 0x3fff0d38
ftpserver	table: 0x3fff1d00
> =node.heap()
25312
> collectgarbage()
> =node.heap()
25312
> for k,v in pairs(_G) do print(k,v) end
pairs	function: 0x3fff0468
__index	table: 0x3ffef548
package	table: 0x3fff0de8
_G	table: 0x3fff0438
_VERSION	Lua 5.1
ipairs	function: 0x3fff09c0
FTP	table: 0x3fff1d00
dofile	function: 0x3fff19d0
require	function: 0x4023b6bc
LFS	table: 0x3fff0aa0
newproxy	function: 0x3fff0b68
loadfile	function: 0x3fff1978
> reg=debug.getregistry()
> for k,v in pairs(reg) do print(k,v) end
1	function: 0x3fff0bc8
2	userdata: 0x3fff1558
3	userdata: 0x3fff0270
4	function: 0x3fff2038
5	function: 0x3fff0510
6	function: 0x3fff1e60
7	function: 0x3fff1fb8
8	function: 0x3fff20c8
tmr.timer	table: 0x40272ba4
stdout	Pipe: 0x3fff1e90
9	userdata: 0x3fff22c0
_LOADED	table: 0x3fff0d08
file.obj	table: 0x40270ea4
stdin	Pipe: 0x3fff04e0
net.tcpsocket	table: 0x40271788
file.vol	table: 0x40270df4
_LOADLIB	table: 0x4026e4f4
onerror	function: 0x4024a384
0	10
net.udpsocket	table: 0x402716a8
mqtt.socket	table: 0x40273124
net.tcpserver	table: 0x402717f8
> -- I now open an FTP session
> 24776	Command: SYST

24704	Command: PASV

22080	Opened data socket userdata: 0x3fff2748 from 192.168.1.200:2121 to 192.168.1.2:54420
21976	Command: LIST

18344	poking sender to initiate first xfer
18992	entering sender
22368	entering sender
24384	Command: TYPE I

24344	Command: PASV

21864	Opened data socket userdata: 0x3fff5e10 from 192.168.1.200:2121 to 192.168.1.2:54422
21584	Command: RETR api.img.4a50ef1

19600	poking sender to initiate first xfer
19600	entering sender
22920	entering sender
21280	entering sender
21280	entering sender
19688	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
19640	entering sender
21288	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
22920	entering sender
22920	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
22920	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
22920	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
22920	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
21280	entering sender
=node.heap()
23656
24160	Command: QUIT
> ftp.close()
Lua error: 	stdin:2: ambiguous syntax (function call x new statement) near '('
> -- oops that should have been FTP, but at least errors are now reported
> FTP.close()
> =node.heap()
39152
> ^]
telnet> Connection closed.

and lastly the FTP session:

$ ftp -p  -d 192.168.1.200
Connected to 192.168.1.200.
220 FTP server ready
ftp: setsockopt: Bad file descriptor
Name (192.168.1.200:terry): 
---> USER terry
331 OK. Password required
Password:
---> PASS XXXX
230 Login successful. Username & password correct; proceed.
---> SYST
215 UNIX Type: L8
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
ftp: setsockopt (ignored): Permission denied
---> PASV
227 Entering Passive Mode(192,168,1,200,8,73)
---> LIST
150 Accepted data connection
-rw-r--r-- 1 terry terry   3233 Jan  1 00:00 HTTPloadfile.lua
-rw-r--r-- 1 terry terry  34676 Jan  1 00:00 api.img.4a50ef1
-rw-r--r-- 1 terry terry  10847 Jan  1 00:00 ftpserver.img
-rw-r--r-- 1 terry terry  10324 Jan  1 00:00 ftpserver.lc
-rw-r--r-- 1 terry terry  18321 Jan  1 00:00 ftpserver.lua
-rw-r--r-- 1 terry terry  12107 Jan  1 00:00 lfs.img
-rw-r--r-- 1 terry terry  22820 Jan  1 00:00 math.img.4a82bd0
226 Transfer complete.
ftp> lcd /tmp
Local directory now /tmp
ftp> binary
---> TYPE I
200 TYPE is now 8-bit binary
ftp> get api.img.4a50ef1
local: api.img.4a50ef1 remote: api.img.4a50ef1
ftp: setsockopt (ignored): Permission denied
---> PASV
227 Entering Passive Mode(192,168,1,200,8,73)
---> RETR api.img.4a50ef1
150 Accepted data connection
226 Transfer complete.
34676 bytes received in 1.99 secs (16.9756 kB/s)
ftp> quit
---> QUIT
221 Goodbye

I am not saying that your issue isn't valid, BTW, but just that I can't immediately reproduce it on my current woking dev version, so doing a quick fix isn't practical as I am in the middle of a work package and don't want to change out environments.

[pnkowl]

But I am getting a bit tired from all of the hours that I am putting in at the mo.

It is my understanding that the team is attempting to PR the dev to master soon. As such, I would think that the team would want end users to be testing dev. Am I mistaken? If something is found, do you want end users to hold them and not share? Is opening an issue the right way to share?

[TerryE]

No, just explaining that I am human and make mistakes, but try to apologise when I do. There isn't some company funding a large dev team here; just one guy putting the hours in unpaid.

[pnkowl]

@TerryE, I have continued from the work you did above. Much appreciated. I will post my follow on work next, but the problem must be intermittent based on unknown factors (e.g. firmware, LFS, Win7, putty, filezilla, command order). These are some of the hardest problems to isolate.

Summary

• Based on your comments just above
• I think it best if I work on getting the development environment on one of my computers.
• My goal is to avoid asking for help in the issues section of nodemcu
• I will refrain from posting nodemcu related things until a word from you (or after the "dev to master PR")
• If you believe there is something that I could do to help nodemcu, let me know.
• The remainder of this post describes why I feel this is the best choice
• Once again, @TerryE, your recent posts help, thank you for taking the time to do all of this.

Is taking a break the most helpful thing?

Maybe it's time to take a few days break and read a book or two.

Honestly, how can I help you?

Creating more tension is not going to move us forward faster.

Can you think of one task that is on your plate, that you would like not to have to do that you can pass on (that will not make more work for you down the road)? Pass it along.

I believe I can appreciate where you are at. At one point in my career, I inherited a pile of code written by a bunch of engineers (yea I was and am one of those too). Fortunately I had a boss who gave me the space to halt development for 3 months in order to rebuild the foundation. I would suspect you do not have that luxury, and that has to be super frustrating.

For me, I am caught between a rock and a hard place. Not because of you, but just because things happen. I have been making various choices to push away or climb over those obstacles, one by one. As such our paths have crossed.

I have not taken the time to set up my own local development environment so that I can both create firmware and LFS images all by myself. As such, I am dependent on 2 volunteers running those services. More on this at the end.

3 weeks ago I was in the middle of a big upgrade and the LFS firmware webservice no longer produced working images for Master. I assumed the webservice was not working for dev either, so I attempted to contribute by opening issues on odd stuff I had seen. That caused some tension.

I recently found out through pure luck that, the LFS webservice did work for dev branch (but still not master). I am becoming more fearful that LFS webservice for master branch will never be fixed. This means that I need to port my existing applications over to the dev branch. This is the first ticket focused on that. Please continue to know that these are not complaints, just statements of what is.

Time for my own development environment?

As a result of the above beliefs, my interest in deploying the entire develpment environment is growing, not becasue I think it is best for the team, but because that appears the only way for me to move forward. I believe Nodemcu benefits by having a bridge from simple interactive Lua (think welcome.lua and blink_a_pin.lua) to the bigger useful apps. In my opinion, that bridge is those 2 webservices.

So the question before me now is what development environment will it be: RPi or Windows 7 or Windows 10 and there seems no obvious choice to me. Win 7 is still my platform of choice. RPi would be pi zero w since I have 2 in stock that are not deployed. I have not stumbled into development environment documentation that was designed for dummies, so the helpfulness of the documentation may dictate my choice. As always, I will attempt to document the steps I take, both for my reproduceablity, but possibly for others. If you have a suggestion, throw it my way.

Too Many Words

I really don't know why it has to take 10,000 words

Not a quote from this thread, but when if it appears that another person in the discussion does not appear to understand what I am saying or I am puzzled by the reply, then the only way I know how to overcome that is to listen more deeply and, as needed, try again (or walk away).

Hope this helps.

[pnkowl]

@marcelstoer What method was used to elevate this to a bug? E.g. were the symptoms confirmed independently or did it just looked like it should be elevated.

[pnkowl]

Steps Taken Based on work/testing by @TerryE above,

• create a new dev branch firmware with default modules + 64K LFS (otherwise defaults) using cloud builder
• download the entire repository and extract the relevant 3 files from dev branch
• create a new dev branch LFS image with these 3 files (_init.lua, ftpserver.lua. telnet_pipe.lua) with cloudbuilder
• flash all this to nodemcu v0.9 hardware
• try @TerryE first sequence above and some others, all panic
• make an binary image of device (4M bytes, uncompressed)
• flash second device, wemos d1 mini, with binary image from line above
• run more iternations and keep better notes, one passes but attempts to repeat fail

Wemos Iteration Summary

panic           panic           okay            panic           panic           panic           panic
4               5               6               7               8               9               10
ftp             wifi            ftp             ftp             ftp             ftp             ftp             
telnet          ftp             wifi            wifi            wifi            wifi            telnet          
telnet connect  ftp connect     ftp connect     ftp (file list) ftp (recon)     ftp (connect)   ftp (filelist)  
ftp connect     telnet          telnet          telnet          telnet          putty fails     telnet (recon)  
panic           telnet connect  ftp filelist    telnet connect  ftp (filelist)  telnet          ftp (filelist)  
                ftp (filelist)  telnet connect  telnet gets odd telnet connect  telnet connect  panic           
                panic           ftp filelist    ftp filelist    ftp (filelist)  ftp (filelist)                  
                                okay            panic           panic           panic           

Command Syntax

node.flashindex'_init'()
require'telnet_pipe':open('myssid','mypass',2323)
require'ftpserver'.createServer("test","12345",true)

Using

• filezilla
• putty

Zip Contains

3 files
1 firmware
1 LFS image (zip and img)
1 device image

200520b.zip

[pnkowl]

@TerryE, your attempts to duplicate the panic are most appreciated even though you were unable to duplicate the issue. You then did additional testing and everything looked okay. Thank you for doing this.

Next Steps

Should I try a third device type (I have at least 3 different models reserved for development: 2x nodemcu v0.9, 4x wemos D1 mini, 4x bare esp-12F). I have posted all the nodemcu stuff **for perhaps others to review ** With help, I could possibly collect some debug info like you shared above if that would help. The problem is unlikely to go away by itself. OR just let this go for now?

[TerryE]

Win 7 is still my platform of choice.

If you want to do your own builds rather than rely on cloudbuilder -- and this also means that you can luac.cross locally as well then you really need some form of Linux (compatible) environment. I did validate the luac.cross make for Cygwin and MinGW, but I am not sure if anyone has the main make working on these. With Windows 7, the easiest way is to use VirtualBox IMO, and there are a number of pre-built Ubuntu server images available such as Ubuntu18.04-3 from here. You may need to use a few apt installs, but you can then PuTTy onto the VM from the PC. You would also need to add Guest additions to the OS (see the instruction on the VBox site) so that you can connect to the ESP through a host USB. Allocating 1Gb RAM is easily enough for a server VM.

Splitting the issues down is really useful. For example is telnet robust on its own? Ditto FTP, or is the combination needed? If you have specific documentation recommendation, then raise your proposal in general terms just to make sure that we generally agree, then draft specific rewording. If you doing this sort of contribution, I am happy to do some 1-1 chitchat. You will find my email addr in the commit logs; drop me an email.

Also be aware from the initial work on the FTP server is that it is very much a subset implemention and it is sensitive to which FTP client you are using -- for example some open two FTP sessions.

PS: I also have D1 minis as my target workhorse. I am fairly confident that this class of error is module variant independent so I would just stick with these.

[HHHartmann]

You could also use Marcel's Docker image. to create firmware images and LFS. If you only need luac.cross to build LFS images you can build a native windows version with VisualStudio. Just used it last week and it works.

[TerryE]

The issue for any Windows developer is the any of these paths is foreign and involves be learning curve. IMO VBox + prebuilt Ubuntu image is just as easy as docker, but each to his own. Ditto Cygwin and MinGW are far smaller installs than VS, but I haven't logged onto Windows in anger in over a decade - and this is from a guy who has been to MS @ Redmond three times because of all the MS dev work my teams used to do. 🤣

[jmd13391]

You could also use Marcel's Docker image. to create firmware images and LFS.

Before LFS, it was @marcelstoer 's awesome ESP8266 Cloud Builder Service all the way for me (a typical NodeMCU end-user). Without the Cloud Builder Service, I would found myself developing on the Arduino long ago. I migrated to the Win7/10-Docker-LFS-esptool-ESPlorer development path since the day LFS went live (thanks @TerryE - we are not worthy - lol). IMO, this is the path of least resistance for developers such as me that live mostly in the Windows universe. I find it only takes me about 2 minutes and 2 or 3 cmd lines each time I have a new myapp.lua variant to test on the ESP hardware.

I do have a workflow process cheat sheet of sorts in txt format I'd be willing to share if anyone is interested. I know this is not the proper forum for such things so PM me if interested.

[marcelstoer]

OT-on

I do have a workflow process cheat sheet of sorts in txt format I'd be willing to share if anyone is interested. I know this is not the proper forum for such things so PM me if interested.

Care to share it through a wiki page at https://github.com/nodemcu/nodemcu-firmware/wiki? I think any GH user can create new pages.

OT-off

[jmd13391]

@marcelstoer -- Windows Docker LFS esptool ESPlorer Workflow Cheat Sheet

[edit:] Content reformat by TE, Page rename by MS

[HHHartmann]

Back to the original issue I can report that I also observed this issue.
What I did:

1. Start telnet and ftp servers
2. List files with ftp (repeated)
3. open telnet session (using on board win 10 telnet)
4. List files with ftp (repeated) Observe output on telnet session
5. type a single letter (e.g. a)
6. observe output like this in the telnet session

attempt to call a table value
stack traceback:
[C]: ?
[C]: ?

7. reload ftp and observe the error in line 121

NodeMCU 3.0.0.0
branch: dev
commit: 50fcd72 (which is the latest commit)
release: 2.0.0-master_20170202 +515
release DTS:
SSL: false
build type: float
LFS: 0x40000 bytes total capacity
modules: adc,bit,color_utils,crypto,dht,encoder,enduser_setup,file,gpio,http,i2c,l3g4200d,mdns,net,node,ow,pipe,pwm,rtcfifo,rtcmem,rtctime,sjson,sntp,spi,struct,tmr,uart,wifi,wifi_monitor,ws2812,ws2812_effects
build 2020-05-17 13:02 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

[TerryE]

@HHHartmann Gregor, can you just confirm that you are using telnet_pipe.lua. As we've discussed above, the other version doesn't work, and should be removed.

[vsky279]

I am using telnet_pipe.lua.

The restart does not occur before PR #3075.

[HHHartmann]

@TerryE Just prepared an LFS with only telnet_pipe and ftpserver to outrule any interference from my usual startup script.
Started it with

node.flashindex('telnet_pipe')():open("wifi","xxxxx",23)
node.flashindex('ftpserver')().createServer("test","12345",true)

Same results

[HHHartmann]

@TerryE just for your Information, you don't need telnet to reproduce.
A simple call like node.input("a") yields the same error message as mentioned above and leads to the same crash. Also encapsulating the call in a timer does not change the result.

[TerryE]

Yup, thanks. Input lines are expected to be CR terminated, so node.input("a\n") should work fine. Need to look at the error paths

[TerryE]

The fix to the telnet issue was a one liner in lua.c - easy fix; bloody hard to track down. I've done this as an immediate commit to dev (8f85c9b) so that this is now available.

@marcelstoer, please note. Thanks

[marcelstoer]

...telnet_pipe.lua. As we've discussed above, the other version doesn't work, and should be removed.

• Will you do this or was this a general remark?
• Should this be done in the context of this issue or can we close it?
• It's my understanding this small fix can still land on dev before the cut?

Btw, I just noticed the file links in /lua_examples/telnet/README.md are broken.

[TerryE]

I can't remember what happened here and why telnet.lua got renamed, breaking the links. @nwf and I agreed that:

• telnet_fifo.lua should be removed
• telnet_pipe.lua should be renamed to telnet.lua
• Given that this is a module that a few developers use, I also think that it makes sense to move the telnet folder to lua_modules and the README.md to the appropriate docs location.
• Yes, I will raise this PR tomorrow, if @marcelstoer you are happy with what I suggest.

[marcelstoer]

👍 SGTM, thanks Terry!

[marcelstoer]

@TerryE I prepared theses changes in #3133.

[TerryE]

@marcelstoer. let me do this, but I've got some other stuff to do this weekend so this might take a few days. Sorry.

[marcelstoer]

Unless you'd like to add changes to telnet.lua (ex telnet_pipe.lua) #3133 is complete AFAICT. It contains what we discussed here.

[TerryE]

@marcelstoer, I suggested that we move telnet into lua_modules so that means the readme needs renaminng and hooking into mkdocs.yml. I also suggested refreshing the ftpserver.lua to fix one of the bugs reported above.

[marcelstoer]

I suggested that we move telnet into lua_modules

That and the necessary renamings are done.

hooking into mkdocs.yml

Bummer, will fix it right away.

I also suggested refreshing the ftpserver.lua to fix one of the bugs reported above.

Ok, I missed that. You can either add that fix to the branch I prepared or merge that one and commit it separately.