Out of date embedded esptool.py on Windows

[Turgut Guneysu]

Hi john,

I was able to successfully load the latest FW onto both WEMOS 8266 and ESP8266 LOLIN that I have. And they both work OK in microBlocks. When I do the update manually, the version that executes is pytools.py ver2.8

However, I tested the “Update Firmware / Download and Install Latest VM” options in the menu and never get a good update and I get errors on the command console:

esptool.py v2.6
Found 1 serial ports
Traceback (most recent call last):
File "esptool.py", line 2959, in <module>
File "esptool.py", line 2952, in _main
File "esptool.py", line 2643, in main
LookupError: unknown encoding: cp65001
Failed to execute script esptool
All tasks stopped
All tasks stopped
Connected to COM7
All tasks stopped
The strange thing is, I do not have esptool.py ver 2.6 on my system. I have two Python versions installed (2.7 and 3.8) an their pytools.py is ver 2.8

So, where does the microBlocks get the pytools ver2.6 and could that be the reason why I do not get a successful update from within microBlocks ?

‌

FINALLY: Bernat’s write up on how to update manually mentions that one should NOT be running microBlocks when doing the pytool.py update, because it conflicts. Yet, the internal update mentioned above happens (or does NOT happen) while MB is running.

Is there a conflict there, and hence my in IDE updates do not work ?

Thx.

[John Maloney]

Thanks for reporting this. Good observation that the versions of esptool.py differ.

MicroBlocks embeds its own copy of esptool.py so users don’t have to find and download it. For Windows, I believe that itself is embedded in a standalone Python app.

However, it appears that the copy of esptool.py we embeded is out of date. We’ll try to find a more recent embeddable version.

FINALLY: Bernat’s write up on how to update manually mentions that one should NOT be running microBlocks when doing the pytool.py update, because it conflicts. Yet, the internal update mentioned above happens (or does NOT happen) while MB is running.

Is there a conflict there, and hence my in IDE updates do not work ?

No conflict. When MicroBlocks itself is attempting to install an ESP VM it suspends its normal communications with the board (which interferes with the data that esptool.py is sending). Once the new VM is installed (if the install succeeds), then MicroBlocks reconnects to the board.

However, if you run esptool from the command line while MicroBlocks is running then MicroBlocks continues to attempt to communicate with the board, causing esptool to fail. (This problem has bitten me so many times it's embarrassing…)

[Bernat Romagosa]

In Windows we’re not using the esptool Python script, but a compiled version that someone built and that is the one the Arduino IDE uses. For some reason, it hasn’t been updated in ages.

I’ll try to find a more recent one, but if a Windows user can try to compile one for us, that’d be great. I don’t have any Windows machines anywhere near me…

The reason for not using the Python script in Windows is that we’d need to ask users to install Python first, and that’s not so trivial for regular Windows users.

[Bernat Romagosa]

I’ve just checked, and the latest Arduino espressif package still comes with an outdated binary esptool. Our choices are:

a) Embed esptool.py and make Python a prereq for Windows users. This is unrealistic, as we’re catering non-techinal users and we can’t force them to install Python in their computers.

b) Embed esptool.py and a portable Python binary for Windows users. This is not unrealistic, but it’d be quite ridiculous to have to include a binary that weighs way more than the whole MicroBlocks environment.

c) Use John’s GP esptool implementation. This one is still in the works, but in the long term it’s the one we should be going for, as it will mean we won’t depend on any third parties.

d) Find a Windows user who can compile esptool.py for us. It’s supposedly as easy as running python -m py_compile esptool.py. @Turgut Guneysu , do you think you could help us with this?

EDIT: Sorry, I just realized that `py_compile` builds Python bytecode, not executable binaries! You’ll need to use something like py2exe instead.

[John Maloney]

Are there instructions on how to build that Arduino espressif esptool package? If it is easy, that could be a good short term solution. Long term, I favor (c) because that would work even on Chromebooks or in a browser-based implementation of MicroBlocks.

[Bernat Romagosa]

That package is just a bunch of Arduino board definitions and (in the case of Windows) an esptool binary. There’s no info about how they built that binary…

[John Maloney]

The error message “unknown encoding: cp65001” is caused by Python not recognizing a UTF-8 string.

https://stackoverflow.com/questions/35176270/python-2-7-lookuperror-unknown-encoding-cp65001

So maybe the problem isn’t that the embedded version of esptool is out of date but rather than the embedded version Python is not handling some Unicode string from Turgut’s environment. For example, it might be getting an error due to a Unicode character in the path to his home directory. That would explain why I haven’t seen this error when testing this mechanism on Windows.

If that’s the problem, a possible work-around would be to create a user account using only 7-bit ASCII for the user name, and use that account to install ESP VM’s. Not very convenient, but perhaps easier than installing Python 3 and esptool. (Python3 has built-in Unicode support, so it shouldn’t get the same error.)

[Turgut Guneysu]

Bernat & John

I tried compiling the esptool.py with no success. I was able to create a version that runs, checks the COM ports, but produces an error:

esptool.py v2.8
Found 1 serial ports
Traceback (most recent call last):
File "esptool.py", line 3201, in <mod
File "esptool.py", line 3194, in _main
File "esptool.py", line 2880, in main
LookupError: unknown encoding: cp65001

When I looked it up, it was due to Python codepage issues in Windows. I tried to use the recommended set PYTHONIOENCODING=UTF-8 command. It did not solve the problem.

They say it does not happen in Python3.8. I will try to uninstall all Pythons and just try it with 3.8.

I’ll update you if anything changes.

[John Maloney]

Thanks for trying this. One of the big changes from Python 2.x to 3.x is that all strings are Unicode strings. I’m hoping that will avoid this issue.

You might also look at line 3201 of your version of "esptool.py". It might be trying to import a module and trying to open a Python module repository in your home directory (just a guess based on “<mod”). If so, there might be a way to change where Python looks for modules to avoid looking in that directory.

[Turgut Guneysu]

OK Guys,

I think I got it. I reinstalled a clean Python 3.8 and used pyInstaller to create the executable.

Here is stuff needed after Python Install:

Install Python Distro

update PIP:
python -m pip install --upgrade pip setuptools wheel

install esptool:
pip install esptool

install pyInstaller to create exe files
pip install pyinstaller

In case you want to try, here is the windows BAT file to create the executable:

Of course, directories need to be adjusted according to the local install details.

rmdir /s /q build_tmp
mkdir build_tmp
C:\Python\Python38\Scripts\pyinstaller ^
    --onefile ^
    --specpath build_tmp ^
    --workpath build_tmp\build ^
    --distpath build_tmp\dist ^
    C:\Python\Python38\Lib\site-packages\esptool.py

‌

I tested and it works on my system.

Here is the command line syntax (with my file locations):

esptool write_flash 0 "D:\PGMDev&Tech__microBlocks\ESP8266_NODEMCU Firmware\vm.ino.nodemcu.bin"

Now, how do I attach an EXE file ???

SENT in email to you and Bernat

GOOGLE DRIVE Link: https://drive.google.com/open?id=1h5AsUD98CDbKWvS6q5LDVqqqn6wcdmbi

[Bernat Romagosa]

Hi, Turgut.

Great to hear you got it working!

Gmail just decided to strip the exe file from your email… Can you maybe upload it to google drive, dropbox or wetransfer?

Thanks!

[Turgut Guneysu]

Hi Bernat

The Google drive link is in Issues.

Get Outlook for Androidhttps://aka.ms/ghei36

[John Maloney]

Thank you very much for figuring this out!

It’s especially helpful to have that BAT file so we can update to later versions of esptool.py in the future. Were does it put the .exe file? (I’m guessing it appears in “build_tmp\dist”?

[Turgut Guneysu]

Yes, to the dist directory.

[Bernat Romagosa]

We’re now testing a custom compiled esptool.exe binary in a separate branch. The next release (not the one from yesterday, but the next one) is going to include it

Thank you, Turgut, for your help once again!