1. John Maloney Avatar John Maloney
  2. all
  3. SmallVM
  4. Issues

Are there any plans to to add MQTT library and math library？

Issue #241 resolved
Wenjie Wu created an issue

Hi @John Maloney

I recently recommended MicroBlocks to many of my friends and they all fell madly in love with it! so I'm forwarding some of their questions:

  1. Are there any plans to to add MQTT library? (Several people strongly want it! )
  2. Are there any plans to to add math library? (A physics teacher uses MicroBlocks to do physics calculations, and wants more math blocks（like Python math library: sin/cos/…）)

‌

Comments (46)

  3. John Maloney repo owner

    MicroBlocks deliberately does not support floating point because it can slow programs down in unexpected ways on microcontrollers without FPU units (most of the ones we support). You can get around that by working with “fixed point” numbers – integers scaled up be a some factor. We have a sine function that works that way. It is used in the turtle graphics library and there is a comment about it there.

    We’ve thought of adding an MQTT library but it is low priority. Is that something you might be interested in writing?

  4. Wenjie Wu reporter

    Thanks!

    I have forwarded your response to them and they thanked you for your reply！

    Is that something you might be interested in writing?

    I'm not sure if I'm capable of doing it : (, but I'll try it.

  5. John Maloney repo owner

    I hope you will have fun trying this, but no problem if you decide to give up. One way to start might be to find implementations for Arduino and read the code. You probably would not do things the same way in MicroBlocks, but reading the Arduino code can help with understanding what needs to be done – assuming the Arduino code is well organized.

    Going to close this issue but free to continue to comment on it on the MQTT topic.

    ‌

  9. Wenjie Wu reporter

    Hi John,

    I'm almost done with the MQTT library and it works.

    it's Chinese New Year holiday recently. When my holiday is over, I will do some optimization on it and create a PR.

  14. John Maloney repo owner

    Nice work! I look forward to hearing how it gets used.

    How extensively have you tested this?

    Do you know how much RAM the library requires when it is not being used? How much extra RAM does it need when running?

    I’ve merged the PR but I will make this feature a VM build option, at least initially. That is, folks who want to use MQTT will need to build their own VM with a compiler flag such as -DMQTT. Anyone who wants MQTT is probably capable of downloading the source code and building the MicroBlocks VM using either the Arudino IDE or Platformio. As you know, that’s not difficult.

    I’m being conservative because RAM is a scare resource. On the ESP32, the MicroBlocks VM is stored in RAM (including all libraries that are linked in), so even if the MQTT primitives are not used, the C++ MQTT library code consumes RAM. When RAM gets too low on the ESP32, then network operations can become unreliable and MicroBlocks itself can crash.

  15. Wenjie Wu reporter

    Hi John, I'm so sorry :(

    I may not be able to answer these questions. I'm an amateur in hardware programming and I've only done a few simple examples.

    If I find any errors in my use, I will let you know immediately.

  16. John Maloney repo owner

    Hi, Wenji.

    No problem! It is difficult to know how much RAM a complex network package needs because they allocate memory dynamically and the amount needed depends on the usage patterns and load.

    In terms of static memory, Platformio’s output suggests that it only consumes an additional 11-12k of RAM when MQTT isn’t used. That’s better than expected.

    I did a quick test and, unfortunately, got an error that crashed MicroBlocks. The sequence was:

    1. connect to my WiFi network
    2. click the “MQTT connect to test.mosquitto.org” block
    3. click the “MQTT connected” reporter block

    The result was a VM hang with the error message:

    [E][WiFiClient.cpp:258] connect(): socket error on fd 54, errno: 113, "Software caused connection abort"

    That message repeats and the IDE loses connection to the board.

    I could be doing something wrong with MQTT.

    After pulling my changes, you can build and install the MQTT VM by doing:

    pio run -e esp32-mqtt -t upload

    Let me know if it works for you. If you also get that error, perhaps we can ask someone familiar with the ESP32 MQTT library to help us debug it.

  17. Wenjie Wu reporter

    hi John, I also encountered this problem before, as you said:

    Perhaps it was down earlier.

    So I updated the broker(you have merged)

    I'm going to add some try/except mechanism in this part

    ‌

  18. Wenjie Wu reporter

    @John Maloney By the way，@Tom Ming said:

    The mqtt library you are using works fine on the esp8266 chip 4M flash

    ‌

  19. John Maloney repo owner

    Good to know. The ESP8266 has much less memory than the ESP32. MicroBlocks requires fair amount of RAM for things like the MicroBlocks code cache. We decided not to support WebSockets on the ESP8266 for that reason.

  20. John Maloney repo owner

    This Google search:

    esp32 "Software caused connection abort" MQTT

    get’s many hits. It sounds as though MQTT on ESP32’s has some issues. This post:

    https://stackoverflow.com/questions/56664817/mqtt-software-caused-connection-abort-error

    suggests that the ESP32 can connect to a MQTT server on the same local network but not a remote one.

    In any case, any errors should be caught and handled gracefully (perhaps using try/except, as you suggest) so that MicroBlocks continues to run smoothly even if the MQTT connection fails.

    ‌

  21. Wenjie Wu reporter

    We decided not to support WebSockets on the ESP8266 for that reason.

    I noticed this, so only add MQTT support on esp32.

    ‌

  23. Simon Walters

    I’ve just found this thread and as soon as I’ve finished getting an MQTT extension accepted into Snap! - I’ll be back to try this out - very excited 🙂

  24. John Maloney repo owner

    @Simon Walters Glad you like it! This is all the work of Wenjie Wu with some help from Tom Ming. Thank you, Wenjie and Tom!

    Note that this is still experimental, so you’ll need to build a custom ESP32 or ESP8266 VM with MQTT enabled. Easiest way to do that is using Platformio to build and install it with this command line line:

    pio run -e esp32-mqtt -t upload

    You can also build for a Wemos D1 mini using d1mini-mqtt in place of esp32-mqtt.

  25. John Maloney repo owner

    I have now included the MQTT primitives in all ESP32 and ESP8266 builds.

    With dynamically allocated buffers, the MQTT code requires only 40 bytes of RAM if the system is not used.

    Buffer sizes can be set to 32 to 16k bytes. The default is 128 bytes. The buffer size is used for four dynamically allocated objects. Setting the buffer size to N will use 4*N bytes of RAM from the heap. Larger buffer sizes should be fine on the ESP32. However, larger buffer sizes could cause instability or failure on the ESP8266. I have not tested this on the ESP8266 but I’d guess that buffers sizes up to 2k would work, 16k would fail immediately, and some of the in-between sizes may be unstable, especially with heavy network traffic (MQTT or other WiFi use).

    If anyone tests on ESP8266, please add a comment here with your results.

    MQTT is a great addition to MicroBlocks. Thank you, Wenji and Tom!

  26. John Maloney repo owner

    I just did some testing on the ESP8266. As expected, it crashes with buffer sizes 2k or larger. It did not crash with 1500 bytes, but it did not work. It worked okay with 1k buffers, although I did only very light testing.

    Based on my quick tests, I’d recommend sticking with buffer sizes of 512 bytes and below on ESP8266 boards. The default buffer size of 128 worked for my test case - publishing the string “Hello!” to the topic “Test topic”.

  28. Simon Walters

    OK -I’m trying to use the blocks - I can connect to my broker and subscribe to a topic but I’m struggling with how to deal with received messages

    I seem to have worked out that MQTT event contains the last “event” (which I’m assuming is the last message received) but reading it seems to clear it.

    So I’m a bit stumped as to how to wait for messages and then take action on them

    ‌

    #Confused Simple Simon :)

  29. Simon Walters

    Re client id

    I’d recommend that the default is blank.

    If its blank, the broker will generate a random id to be used for the session. Which is a good approach if you have several devices connecting to the same broker.

    If you use the same client id on several devices, then it will usually cause issues.

    Brokers will tend to deny access to the second device using same id or disconnect the first one

    This is my understanding anyway

  31. John Maloney repo owner

    Glad you worked it out. You’re correct, the “last MQTT event” block clears the message, if any, after reading it.

    I’m new to MQTT myself, so thanks for tip about client ID’s. I will change the default to the empty string.

  32. Simon Walters

    I wanted to have neopixels light up with current cheerlights colour on a WEMOS D1 mini (ESP8266)

    I mean - who wouldn’t? :)

    But, there seems to be some sort of conflict when the Neopixels and MQTT libs are both loaded

    This script works find without MQTT lib but there are issues when it tries to display the blue if MQTT lib loaded in

    ‌

    PS can this issue be changed back to Open please?

    [edit] Actually seem to be getting issues even without the MQTT lib being loaded - investigating further

  33. Simon Walters

    OK - slowing down 🙂

    Tried using standalone main version

    ‌

    Seems to be rock solid - all works as expected

    Switched back to using pilot version (but still running v120 firmware - all OK)

    Updated firmware and inconsistent behaviour re-appeared

    ‌

  34. John Maloney repo owner

    Thanks! This could be a pin mapping issue, especially if you are using an ESP8266 board. I recently added a mechanism to use the pin numbers marked on ESP8266 boards, which are different from the internal pin numbers. Try using the GPIO pin number from the pinout for your board (see pinouts). Let me know if that works. If it is a pin mapping issue, I will fix it in the next Pilot release.

  35. Simon Walters

    So I’m connected to the pin marked as D4 on the board and setting pin 4 in Microblocks as shown above

    ‌

    According to online images D4 is actually GPIO 2 - if I change Microblocks to pin2 - nothing happens at all

    Just to clarify - I am getting the neopixels to light up - but not in the colours I’m requesting 🙂

    ‌

    I’ve also tried connecting to pin D1 and same result

    ‌

  36. John Maloney repo owner

    Okay, not a pin issue, then. I’m traveling now; I’ll check it out when I get home next week.

    Does the problem have anything to do with MQTT? Sounds like it doesn’t…

  37. Simon Walters

    Nothing to do with MQTT - I just thought it was to start with. Not got anything loaded apart from neopixel lib

  41. Markus Gälli

    Ok, thx. Got it working without my extension on Adafruit though, and the extension also doesn't work against other services.
    Guess I should ask (myself): Why should the secure connection work without ever having put a certificate on the esp32 before? And: Is that a good idea anyhow?

  42. Markus Gälli

    I found the manual on Adafruit confusing as well, took me a while to understand the expected format of pub messages.
    Succeeded with the following, but didn’t try anything else yet:

    ‌

  43. Simon Walters

    On big issue with Adafruit is that if you make a mistook - it will treat you as a hacker and lock you out for a while so it doesn't lend itself to testing :)

  45. John Maloney repo owner

    Another problem with Adafruit's broker is that they limit the update rate. That makes it unsuitable for watching data that changes multiple times per second. Some of the other MQTT servers Simon mentioned don't have that restriction -- but they also don't support the nice UI widgets that Adafruit supports.

    I think you need do something about certificates to support secure connections. When I looked into that a few years ago it looked both more complicated and more RAM intensive, which would be a problem on the ESP8266 which has surprisingly limited RAM. But I'd happy to be proved wrong about that...

  46. Simon Walters

    In other news, My WEMOS D1 mini using the MQTT library to receive Cheerlights and display them on a Neopixel strip had been running flawlessly for over 2 weeks :)

    ‌

  47. Log in to comment
Assignee
–
Type
bug
Priority
major
Status
resolved
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!