Thanks to visit codestin.com
Credit goes to github.com

Skip to content

"Integrated handling for filesystem and gzipped Binaries" (Restart with fresh Fork) #8338

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 0 commits into from
Closed

"Integrated handling for filesystem and gzipped Binaries" (Restart with fresh Fork) #8338

wants to merge 0 commits into from

Conversation

gneiss15
Copy link
Contributor

ATENTION: This is tested on Linux only! See below for open questions/prerequisites.

Currently there are some manual steps necessary to prepare the files necessary for OTA updates (especially for ESP8266HTTPUpdateServer)

The steps are:

Compile the sketch
Export the generated sketch-binary
Generate the filesystem
Export the fs-binary
Generate gzipped versions of binaries
Generate signed versions of gzipped binaries
Which of the above steps are necessary depends on the sketch (does it use a fs?) and the kind of OTA chosen (signed binaries necessary?) or preferred usage (use of gzipped binaries for faster upload?).

There are some problems/inconveniences:
2. The signed sketch-binary is not exported when using "Export compiled Binary".
(This point will be solved with my Pull-Request #8255)
3. & 4. AFAIK, the only “integrated” way to generate the fs is using one of the tools:
“arduino-esp8266littlefs-plugin” or “arduino-esp8266fs-plugin”.
Both tools are designed to create & upload the fs, but not export them.
So if preparing the files for an OTA when no board is connected, you have to:

ignore errors shown because upload is not possible
find & navigate to the “build.path” and copy the fs-binary manually
& 6. AFAIK, there is no “integrated” way to perform these steps.
This Pull-Request will integrate all of the above steps into the “compile and/or upload” action of Arduino.

This is done by adding 3 new entries (“Upload”, "Filesystem" and "Export") to the tools-menu.
These entries are presented just like any other options for esp8266 (“Upload speed” ... “Non-32-Bit Access”) and placed right after the last.

Below is the text that is already integrated into "doc/filesystem.rst"
--- schnipp ---

Menu-Entry “Upload” lets You choose between:

  • Sketch
  • Filesystem
  • Both
    This Menu controls which parts are uploaded when "Sketch->Upload" (Ctrl-U)
    is selected from the Arduino Menu.

Menu-Entry “Filesystem” lets You choose between:

  • Off
  • LitteFs
  • SPIFFS
    This Menu controls which Fs will be created when "Sketch->Compile" (Ctrl-R)
    is selected from the Arduino Menu. The Fs will always be created inside the
    "export" dir (see below).

Menu-Entry “Export” lets You choose between:

  • Off
  • .bin & .bin.signed)
  • Create & Export gzipped Binaries too
    This Menu controls export & generation of (extra) Sketch-Binaries.

The "export" dir
................
After a (successful) compile, the files will be exported to a subdir
“bin/{variant}” of Your sketch directory
(“{variant}” replaced with the name of the board as shown behind
“Tools -> Board:”).
Signed variants of all Binaries are generated similar to the
“automatic signing” done for normal sketch binary.

As valid for the "old" tools mentioned below, it is nessesary to place
files you want to be inside the generated file system into a directory
named data inside Your sketch directory.
--- schnapp ---

Open questions/prerequisites:

  1. As stated above I had tested this on Linux only.
    Hopefully some cant test it under windows..

  2. tkinter should be available.
    if it is not, and creating of fs is requested AND the data dir is empty, I didn't know how to ask the user: “Are you sure you want to create an empty ... image?”.
    So in this case a message is given and the fs will not be created.
    (So its not a real problem if tkinter isn't installed)

An additional question for developers:
As mentioned inside function main of “postbuild.py”, there is a strange behavior of the arg "{build.path}" supplied via “platform.txt”. If someone can explain me why this “string” behave as described in the comments, I would be happy ;-)

@gneiss15
Copy link
Contributor Author

gneiss15 commented Oct 14, 2021 •
edited
Loading

As agreed in my old (now closed) PR #8266, I had remade it by starting with a fresh fork of esp8266/Arduino.
After creating the new fork, I reintegrate MY changes from PR #8266 and do some tests here (all work).
So I hope this is now ready for integrating.

@d-a-v Can You please review this ?
@earlephilhower Can You please review this ?

@d-a-v d-a-v added this to the 3.1 milestone Oct 27, 2021
@d-a-v
Copy link
Collaborator

d-a-v commented Nov 1, 2021 •
edited
Loading

Sorry for taking so long to check on your work, I'm just not used to using filesystems flashing (I prefer to use network protocols and download files on first run). I have been trying the "alpha" release on a dummy linux account that I use for testing.
It appears that mklittlefs is not found after installing.
mklittlefs
In platform.txt python3 for example is accessed through {runtime.tools.python3.path}/python3 so mklittlefs should be accessible with {runtime.tools.mklittlefs.path}/mklittlefs. Maybe this variable could be passed to upload.py?

Edit

This is also true with xtensa-gcc path.
FWIW Here's part of the content of the json installer. mklittlefs/spiffs are next to python3 and gcc

               "toolsDependencies": [
                  {
                     "packager": "esp8266",
                     "version": "3.0.4-gcc10.3-1757bed",
                     "name": "xtensa-lx106-elf-gcc"
                  },
                  {
                     "packager": "esp8266",
                     "version": "3.0.4-gcc10.3-1757bed",
                     "name": "mkspiffs"
                  },
                  {
                     "packager": "esp8266",
                     "version": "3.0.4-gcc10.3-1757bed",
                     "name": "mklittlefs"
                  },
                  {
                     "packager": "esp8266",
                     "version": "3.7.2-post1",
                     "name": "python3"
                  }
               ],

Their path shouldn't be guessed (it's actually ~/.arduino15/packages/esp8266/tools/mklittlefs/mklittlefs and the core is at ~/.arduino15/packages/esp8266/hardware/esp8266/0.0.2/{cores,library,tools,...})

@gneiss15
Copy link
Contributor Author

gneiss15 commented Nov 6, 2021 •
edited
Loading

@d-a-v:

  1. mklittlefs will be "pulled in" on a install "Using git version" (see https://github.com/esp8266/Arduino) of the esp8266 toolchain only after You "Initialize the submodules" as described here:
    https://arduino-esp8266.readthedocs.io/en/latest/installing.html#using-git-version

  2. I resolve the path to the FS-build-tool (mklittlefs or mkspiffs) with the following code (inside postbuild.py):
    mkFsPath = Which( mkFsName, os.path.join( ToolsDir, mkFsName ) )
    if mkFsPath is None:
    mkFsPath = Which( "%s.exe" % mkFsName, os.path.join( ToolsDir, mkFsName ) )

This assumes that these tools are located below "{runtime.platform.path}/tools".
With this assumption the path to both fs-build-tools (on linux and window) should be correctly resolved.
It will result in mkFsPath being None if the tool wasn't installed (as it must be the case on Your test).

The message You see is a bit confusing, because the second parameter that should indicate the path were the tool was searched was chosen wrong.
That's why You see a "None" after "not exist:"
And Yes, these check and resulting error messages need a correction. I will commit these ASAP.

But as long as You had done the "Initialize the submodules", the code will resolve the path to the FS-tool correctly...

  1. The approach shown above may fail if You had installed these tools at a different place (but platform.txt has correctly been adopted).
    So Your approach of using the path from platform.txt is definitely better.

This will require:

  • add two extra parameter (tools.mkspiffs.path and tools.mklittlefs.path) to "postbuild.py" (that, not upload.py is the part that handles FS creation)
  • I then still need to try if we are running on linux or windows....
  • so the above code (using Which) is still preffered
  • OK I will try to implement all of the above..

OK, I had implemented these changes and will commit them hereafter
With this new commit You will get the correct message if You haven't installed the selected FS-tools

@gneiss15
Copy link
Contributor Author

gneiss15 commented Nov 6, 2021

Sorry, Commit text is wrong it should have been "Improved..." not "Aproved..."

@gneiss15
Copy link
Contributor Author

gneiss15 commented Dec 5, 2021

@d-a-v
@earlephilhower
Almost 30 Days without any reaction ?

@d-a-v
Copy link
Collaborator

d-a-v commented Dec 5, 2021

#6690 is mine, is more than 2 years old, and hasn't received any public comment since 9 months.

Please be patient, we don't have more free time than you do.

@gneiss15
Copy link
Contributor Author

gneiss15 commented Dec 6, 2021

@d-a-v That sound like You haven't enough manpower. May I help anywhere ?

@d-a-v d-a-v modified the milestones: 3.1, 4.0.0 Dec 16, 2022
@d-a-v d-a-v added merge-conflict PR has a merge conflict that needs manual correction and removed alpha included in alpha release labels Feb 8, 2023
@gneiss15 gneiss15 closed this Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
component: OTA merge-conflict PR has a merge conflict that needs manual correction type: enhancement
Projects
None yet
Milestone
4.0.0
Development

Successfully merging this pull request may close these issues.
2 participants
@gneiss15 @d-a-v