LFS used/free size (Feature requires)

[chathurangawijetunge]

• How to check if existing img is same or equal to new one before updating to LFS
• Ways to get available space/used in LFS
• Compare existing LFS img with new img

[KT819GM]

+1 for this, also seeking a way of properly hashing lfs images to make some auto-lfs-ota trough http/https.

[TerryE]

Added in #3193. node.info'lfs' now reports both the LFS region size and used.

The Unix timestamp for the LFS compile is recorded in the LFSheader. Post #3193 this can be accessed by the property node.LFS.time. In practice this should be a unique ID for a given LFS image.

[HHHartmann]

This would be really useful if there was a way to extract this timestamp from an image file as well. That way I can easily decide whether I have the latest image installed or not.
Lately I observed, that flashing fails after a considerably longer time than a successful flash.
The boot reason says that there was a WD timeout.

[TerryE]

For Lua 5.3, the timestamp is the last integer in the image. See lundump.c functions LoadAllProtos() and LoadInteger(). Integers are variable byte encoded (because most integers in a dumpfile are small ones). In the case of an integer in the 1,500M range, this is the concatenation of the least sig 4,7,7,7,7 bits of the last 5 bytes in the file, so a botch method of getting this is:

function getTS(image)
  local f=file.open(image)
  if f and f:seek('end',-5) > 0 then 
    local t = f:read(1):byte()&0xf
    for i = 1,4 do t = (t<<7) + (f:read(1):byte()&0x7f) end
    f:close()
    return t
  end
end

Note this is Lua 5.3 code. It's more difficult for Lua 5.1 since the image is gzipped.

Lately I observed, that flashing fails after a considerably longer time than a successful flash.

I've optimised the load process somewhat. The Lua runtime just closes and restarts rather doing a CPU restart for the intermediate step and does a fast restart for the last restart. Simple failures (e.g. the file doesn't exist) don't restart at all, but rather just return an error. I can look into this further if you raise a specific test case and issue.

[HHHartmann]

Thanks for the sample code for retrieving the timestamp. Will use it once I switched to 5.3.
For 5.1 there is no node.info about the installed LFS's timestamp, so I don't need it there.

The failing flash I observe on Lua 51. When I can find a testcase I will post an issue. It always worked on the second try so far. Maybe it is also connected to a not freshly booted system.

[HHHartmann]

Reading the original request, the above Lua sample to get the timestamp just mentioned in this issue feels incomplete.
I would love to see a node.LFSfileinfo(filename) method returning information about the content of the file. At least the timestamp(LUA53 only of course) Maybe other information like a list of names of the modules. But I am really not sure if this is needed.

@TerryE I leave it to you if you want to reopen this issue.

[TerryE]

@HHHartmann, I am about to start the issue / PR for adding a second LFS and the ability (in Lua 5.3) to build LFS incrementally on ESP. Part of this understanding the dump format and its grammar. This an encoded byte stream that has a left-to-right grammar and must to be parsed start to end. For historic convenience, the timestamp is the last integer value in the file, and the only way I can quickly decode this is a botch; anything else with the current format would be a full parse of the file ignoring anything but the timestamp. This just isn't worth it in terms of effort.

What I suspect is that introducing this incremental build capability is going to introduce a break in LFS image format anyway, (that is will require any images to be recompiled to be loadable with the new loader). I feel that using the Unix timestamp is not a portable ID when doing builds on the ESP, and that the functional requirement is that you should be able to specify some form of unique ID or equivalent for the image and the ID should be in the image header, and also the image should have a CRC (as is the case implicitly with the gzipped Lua 5.1 format).

We should be able to do some form of node.LFS.verify(imagename) which returns the ID of the image if valid and node.LFS.id which returns the ID of the currently loaded LFS. I think that the current standard UUID format is probably overkill but a 32bit value encoded as 8 hex digits or 6 base 64 ones should be adequate. However, let us have this discussion on the new issue, rather than here.

I would also note that my provisioning strategy (see this gist) fetches images from a (local) web service and it uses the file's Etag as a unique ID.

There is nothing to stop you implementing a workaround using a similar strategy and that adds the timestamp to the image filename. (You can rename a plain imgName to '%s.%u' % {imgName, node.LFS.time}). Also file.list"^XXX%.img.*", for example, returns an array of all files starting with XXX.img that you can parse.

[HHHartmann]

@TerryE The problem with the proposed workaround is, that I can only read the node.LFS.time after flashing it.
So when I flash an Image, I don't know which ID (timestamp as of yet) it has.
I always need to store the previous ID on SPIFFS (or wherever) and the compare to the new current one to guess that flashing succeeded or not.

The problem is that I cannot know if a certain file is the one which is flashed or not.
But as workaround it is OKish.

We should be able to do some form of node.LFS.verify(imagename) which returns the ID of the image if valid and node.LFS.id which returns the ID of the currently loaded LFS. I think that the current standard UUID format is probably overkill but a 32bit value encoded as 8 hex digits or 6 base 64 ones should be adequate. However, let us have this discussion on the new issue, rather than here.

That sounds perfect!