1. John Maloney Avatar John Maloney
  2. all
  3. SmallVM
  4. Issues

Allow installing custom ESP32 VM's from stand-alone apps

Issue #497 open
Simon Walters created an issue

Uploaded my firmware.bin file to

https://github.com/cymplecy/fivebyfive/raw/microblocks/microblocks/firmware.bin

and got this when I tried to install it via URL method

‌

Comments (45)

  1. John Maloney repo owner
    • changed status to open

    Thanks for testing this.

    The download is failing for some reason in the IDE, although it works when I click on the link manually. (It should check for that and tell the user.)

    I put the file here:

    https://microblocks.fun/mbtest/tmp3/simon_firmware.bin

    I was able to download that okay.

    I'm guessing that maybe Github needs some sort of credentials for the download to work??? The error message in Chrome says the download failed but not why.

  3. John Maloney repo owner

    Long ago, the standalone version did not support SSL but I added the curl library so now it does. In the browser, we use the browser's HTTP(S) mechanism, so we can use SSL (the link to the file on microblocks.fun is an https link, which works).

    So I'm guessing it is something specific to github. Maybe it needs a cookie or something to know that you are logged in?

  4. John Maloney repo owner

    Or it might be something simpler, such as a timeout that is too short.

    It would be great to make this mechanism work with github.

  5. Simon Walters reporter

    Just tried using your link and it appeared to work but board isn’t respoding to request for firmware version and although I can load a stektch in - it doesn’t run

    ‌

    Tried again by loading a simple Arduino sketch to wipe it and then reloaded firmware via URL - same effect

    ‌

  6. Simon Walters reporter

    After seeming to install the VM - it sits on the connecting screen for quite a while

    ‌

  7. Simon Walters reporter

    Just ran MB from the terminal and I can see that its still running my simple Arduino sketch

    ‌

  8. John Maloney repo owner

    After seeming to install the VM - it sits on the connecting screen for quite a while

    That means that it wasn't able to connect to your board. You may need to manually put it into boot mode by holding the "boot" button while clicking the reset switch.

    On my ESP32 board, it's sufficient to hold the "boot" button while MicroBlocks is trying to connect. Once you xx% you can release the button.

  9. John Maloney repo owner

    When this works you'll see messages in the console reporting that it has installed four files on the board. If you don't see those messages, then the install failed. In this case, it probably timed out while trying to connect to the board.

    When it successfully installs, you'll need to reset the board (reset button or by power cycling) to get it out of boot mode. For some reason several of the new boards do not support the ESPTool message asking them to reset. That happens with Espressif's version of ESPTool as well. The workaround is easy -- just click the reset button when the install has completed, then reconnect the IDE to the board.

  10. Simon Walters reporter

    I’ve managed to get it to load :) … but only once in 10 attempts :( I can’t seem to repeat the feat :( JFI the pio uploader doesn’t need me to press any buttons and the Arduino IDE doesn’t require me to press any buttons

    Trying some more so see what sequence of button pressing works :)

    ‌

    [edit] after many more attempts, I’ve never managed to repeat my one success

  11. Simon Walters reporter

    Just as an experiment - I installed esptool using pip install esptool and used it to upload and it all seemed to work without any button pressing 

    simon@simon-HP-Desktop-M01-F1xxx:/media/simon/DATA/smallvm$ esptool.py --port /dev/ttyACM0 write_flash 0x10000 .pio/build/esp32-c3/firmware.bin 
esptool.py v4.7.0
Serial port /dev/ttyACM0
Connecting...
Detecting chip type... ESP32-C3
Chip is ESP32-C3 (QFN32) (revision v0.3)
Features: WiFi, BLE, Embedded Flash 4MB (XMC)
Crystal is 40MHz
MAC: 84:f7:03:a0:fa:e4
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x00010000 to 0x000f9fff...
Compressed 955040 bytes to 569682...
Wrote 955040 bytes (569682 compressed) at 0x00010000 in 7.9 seconds (effective 967.1 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
simon@simon-HP-Desktop-M01-F1xxx:/media/simon/DATA/smallvm$

    ‌

  12. John Maloney repo owner

    I must have gotten the C3 confused with the S2 or S3. At least one of those gives a warning that you have manually reset the board after installing firmware. Will check which one when I get a chance.

    I was guessing that the download was timing out for you most of the time. However, it doesn't look like there is a timeout.

    Is it getting to the "Connecting" message? If so, I think the download is completing and the issue involves connecting to the board. It has always been a bit tricky to get ESP boards into boot mode automatically. ESPTool (and MicroBlocks) use the serial handshaking lines to do that, but timing issues that make the process unreliable on some boards. It can also depend on your OS, whether ESPTool is running in Chrome or stand-alone, and even whether you are using a USB hub or not.

    Holding the "boot" button while MicroBlocks attempts to connect can make the process more likely to succeed by keeping the BOOT pin asserted while ESPTool resets the board.

  15. John Maloney repo owner

    Which S3 board are you using? I tested with an S3 devkit board, which has two USB connectors. Only one of them connects to the IDE (and hence allows firmware updates).

    You might also try holding down the boot button as soon as the Connecting message appears. In need to do that on one of my ESP32 boards.

    I put MicroBlocks VM's and simple "blink" sketches for S2, S3, and C3 boars in this folder:

    https://microblocks.fun/mbtest/tmp/

    Folks can use these for testing. The S2 ones are for an Adafruit Metro S2, which may have the user LED on a different pin than a generic ESP32-S2, so blink may not do anything but if you succeed in installing it the virtual machine should be able to connect to the IDE.

    Thanks for testing!

  16. Dariusz Dorożalski

    This Github URL responds with the

    HTTP/1.1 301 Moved Permanently Content-Length: 0 Location: https://github.com/cymplecy/fivebyfive/raw/microblocks/microblocks/firmware.bin

    So the "Content-Length: 0" and internal GP routine "download u r l" break at the progress indicator => byteCount(data) / byteNedded. The HTTP client, used by GP, seems to be minimal, handcrafted at the socket level - rather not able to handle 30x redirections.

    Simon, maybe you should serve your file by the hosting service i.e. github.io URL.

  20. Dariusz Dorożalski

    Raw URL of the resource is https://bitbucket.org/cymplecy/smallvm/raw/d7cc546d125a479de660c88ca8f1e09c9ce4d4e7/extraVMs/esp32c3fh4rgb.bin

    This https://bitbucket.org/cymplecy/smallvm/src/dev_cymplecy/extraVMs/esp32c3fh4rgb.bin is BB GUI for "esp32c3fh4rgb.bin"

    BTW. There is a GH hosting for your repo (GH Pages). You can directly serve files from GH i.e. https://dardoro.github.io/firmware.bin

    https://pages.github.com/

    For BB https://support.atlassian.com/bitbucket-cloud/docs/publishing-a-website-on-bitbucket-cloud/

  22. John Maloney repo owner

    Hi, Simon. I just made a quick change (to the webapp only!) to decrease the baud rate.

    Could you try "install ESP firmware from URL" using the pilot version running in a Chrome or Edge browser? Be make sure to shift-reload the page so that you are running version 1.2.71. Then try to install from URL using:

    https://microblocks.fun/mbtest/tmp3/simon_firmware.bin

    Does that work for you?

    Also, can you remind me which S3 board you have or, even better, provide a link for it? I want to see what USB-serial support chip they are using.

  23. Simon Walters reporter

    “Could you try "install ESP firmware from URL" using the pilot version running in a Chrome or Edge browser?”

    How do I run the pilot version in a browser?

    I’ve a cloned copy of your repo and I’ve re-synced it to latest version and recompiled the IDE by running ./build.sh and then launched ./apps/ublocks-linux64bit and it says version 1.2.71 but still unable to connect unfortunately :(

    I’m trying this out on an ESP32-C3 https://uk.banggood.com/ESP32-C3-Development-Board-RISC-V-WiFi-Bluetooth-IoT-Development-Board-Compatible-with-Python-p-1914005.html?cur_warehouse=CN

    ‌

  24. John Maloney repo owner

    To run the pilot IDE the browser, open:

    https://microblocks.fun/run-pilot/microblocks.html

    Required Chrome or Edge in order to access the serial port.

    However, if it did not work in the 1.2.71 standalone app then guessing it won't work in the browser either. However, if you open the Javascript console in the browser it may show some message that will provide a clue.

  25. John Maloney repo owner

    Looking at the pictures of the board I see a button marked "B". Have you tried holding that button down while MicroBlocks is trying to connect to it? If the firmware installer sharts to show progress you can release the button.

  26. Simon Walters reporter

    “However, if you open the Javascript console in the browser it may show some message that will provide a clue.” No, just says Could not connect. Board did not respond. :(

    “Have you tried holding that button down while MicroBlocks is trying to connect to it? If the firmware installer sharts to show progress you can release the button.” Yes - that’s what I’ve been doing - the connect spinner never changes - just keeps spinning :(

  27. John Maloney repo owner

    Hmm. I'm not sure why things are working on my C3 but not on your board. I'll look at the code and try to figure out what else (besides the baud rate) could be making it fail.

    Worst comes to worse I can buy one of those boards, but it looks like there is a long shipping time from China (mid April).

  28. Simon Walters reporter

    I just received another one - unfortunately not all the pixels light up :( I ordered it last Tuesday so it only took a week to get here

    What C3 board do you have and I’ll try and get one of yours as well

    ‌

  29. Simon Walters reporter

    But now they are all working again - must have been a little glitch in the matrix - pun unintended :)

  30. Simon Walters reporter

    JFI As I’ve said above - the file hosting isn’t the main problem that I have but I’ve tried invoking every spell I can come up with to upload the file to a website and get a working link and I have spectacularly failed :) I even ended up borking my main website simplesi.net (while editing .htaccess to try and allow CORS access) and had to get my hosting company to fix it for me :) The only upload location (apart from https://microblocks.fun/mbtest/tmp3/simon_firmware.bin ) that I can get to work was to run up a simple webserver on my Pi5 and serve it up that way http://raspberrypi5.local/firmware.bin :)

  31. John Maloney repo owner

    Sorry you messed up your website trying to allow CORS access. Glad you got it working again.

    You're testing with the stand-alone version, right? You might have better luck using the webapp in the browser since it uses the browser HTTP fetch mechanism.

    The board I have is a ESP32-C3-DevKitC-02:

    https://www.adafruit.com/product/5337

  32. Simon Walters reporter

    Ordered one from AliExpress

    And I’ve been playing around with some of the timing delays in ESPTool.gp but not got anywhere

    I did once get another successful connection but that was by madly pressing the button as fast as I could - it did connect and looked as if pushed the file onto the board but the board didn’t work properly afterwards

  34. Simon Walters reporter

    All I need to do now is to find out where I can host the VM for other people to access to avoid having to send it to you everytime I do an update

  35. John Maloney repo owner

    You may be able to host on Github or Bitbucket if you can get the URL for the raw resource. Did you try the link to the raw .bin file that Dariusz posted?

    He had a few other suggestions about ways to host the .bin files.

  37. Simon Walters reporter

    PROGRESS :) Now that I know how to get the board into bootloading mode - I retried using https://microblocks.fun/run-pilot/microblocks.html with one of Darius links and it downloaded and installed :)

    Going to experiment further to see if it works with standard web version and one of my links

    One of the interesting things is that downloads take a noticable time to execute when running the standalone app on my computer but they just take < 1 sec using web app. Which coupled with the connection issue made me think the download wasn’t working.

  38. Simon Walters reporter

    Summary:

    Using https://microblocks.fun/run-pilot/microblocks.html - putting my ESP32C3 into boot mode using buttons - connecting via connect icon - using install ESP firmware from URL and using one of my github links https://raw.githubusercontent.com/cymplecy/fivebyfive/microblocks/microblocks/firmware.bin (or ones from Darius) works :-)

    Using standard https://microblocks.fun/run/microblocks.html - doesn’t work - the connect icon flashes (about 1 every 2 secs) and VM doesn’t install

    Using standalone app when trying to connect to any (that I’ve tried) https://… link doesn’t work except for https://microblocks.fun/mbtest/tmp3/simon_firmware.bin - I’d love to know why/what/how but it’s not critical anymore :)

    Using standalone app connecting to a local LAN, plain HTTP server works

    ‌

  39. John Maloney repo owner

    Thanks for the summary! I'm glad that you found a way to get MicroBlocks firmware onto you C3 board and even found a few options for hosting it.

    Using standard https://microblocks.fun/run/microblocks.html - doesn’t work - the connect icon flashes (about 1 every 2 secs) and VM doesn’t install

    That's as expected. The stable release does not have the new code needed to support uploading to the ESP-S2, -S3, and -C3 boards.

    Using standalone app when trying to connect to any (that I’ve tried) https://… link doesn’t work except for https://microblocks.fun/mbtest/tmp3/simon_firmware.bin - I’d love to know why/what/how but it’s not critical anymore :)

    I suspect the stand-alone app may not be handling HTTPS correctly. Or maybe it's not handling HTTP forwarding. EIther way, the stand-alone app has a very crude HTTP fetch mechanism compared with the web app (which user the mechanism built into the browser).

    I'll investigate. FYI, which OS are you running on?

  40. John Maloney repo owner

    Okay, investigated. The issue is, as Dariusz said, that the stand-alone version uses a simple socket connection to port 80, which means it can ONLY handle HTTP, not HTTPS.

    This is the case for the stand-alone app on ALL platforms.

    Perhaps we'll eventually update the stand-alone VM's HTTP capabilities. Meanwhile, you can only use the "update ESP firmware from URL" over HTTPS links from the webapp. However, once the firmware is installed on your board, you can use the stand-alone app to program your board with MicroBlocks.

  43. Simon Walters reporter

    As a proof of concept, I’ve come up with this modification that allows the stand-alone app to load from a local file

    method downloadURL MicroBlocksFlasher url {
    // Return the binary data for the given URL or an empty binary data if the download fails.
    // Check if local file
    i = (findSubstring 'file://' url)
    if (not (isNil i)) {
        data = (newBinaryData)
        localfile = (substring url (i + 7))
        // localfile = 'file:///media/simon/DATA/smallvm/.pio/build/esp32-c3/firmware.bin'
        data = (readFile localfile true)
        if (isNil data) { return (newBinaryData) }
        return data
    }

    The idea is that anyone could just use their browser to download a VM to their computer and then install from there

  44. John Maloney repo owner

    Nice! I was trying to think of a way to read a local file without adding yet another menu command. This does that.

    Unfortunately, this mechanism won't work in the browser since Javascript cannot directly read a file from the user's computer for obviously security reasons. It may be simplest to just add another menu item, "install ESP firmware from file".

    Will consider...

  46. Log in to comment
Assignee
–
Type
bug
Priority
minor
Status
open
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!