1. John Maloney Avatar John Maloney
  2. all
  3. SmallVM
  4. Issues

HTTP server can not work with Squeak WebClient

Issue #287 new
Wenjie Wu created an issue

I use Squeak(6.1) WebClient to send a post to HTTP server: WebClient httpPost: 'http://192.168.1.7' content:'hello' type: 'text/plain'.

I can get the right content length(5), but can not get the body of the request(get nothing)

‌

when I use curl , it work well.

curl -X POST -H "Content-Type: text/plain" --data "hello" http://192.168.1.7

‌

‌

‌

this is my code

‌

‌

‌

Comments (29)

  2. John Maloney repo owner

    I'm guessing that curl sends both the headers and body in a single TCP/IP packet whereas Squeak breaks it into several packets.

    I believe you can call:

    scriptImage27120653.png

    a second time (or perhaps multiple times) to read the body data. Let me know if that works.

  3. Wenjie Wu reporter

    When I call the request block manually.

    a first time can read the body data(It is stable).

    The results of manual click and assignment operations are not the same, which makes me wonder.

    ‌

  4. John Maloney repo owner

    I think that is a timing issue. You might try waiting 100 msecs then concatenating the result of a second call to the HTTP server request block to the original data. Let me know if that helps.

    The library should probably be collecting the entire request, including the body of a POST, so users of the library don't need to do that manually.

  6. John Maloney repo owner

    Hmm. Maybe increase that 100 msecs to 1000 or even 5000?

    It is very strange that you get the entire request, including "hello", when you invoke the block manually but not in the server loop. That might be explained by a socket queuing delay on the Squeak side. But I would have expected 100 msecs would be long enough for any queued data to get sent...

    Another possible test: Do a post with a much larger body (say 10k of text). I would expect at least part of that data to be transmitted, even if there some queuing.

  7. Wenjie Wu reporter

    When I increase that 100 msecs to 200 , it works well.

    But I am still confused as to how I was able to get all the data intact every time I manually clicked on the request block, click also in a split second

  8. John Maloney repo owner

    Interesting that the delay must be more than 100 msecs.

    Although I don't know how the Espressif TCP/IP stack actually works, my guess is that it collects data in an internal buffer. When you do a request from Squeak, then manually click the HTTP server request block, enough time has elapsed for the entire HTTP request to get buffered, even if it was transported in several TCP packets. However, when the HTTP server request block is called in the server loop, it returns as soon as the first TCP packet is received, even though the entire request has not yet been received.

    In theory, you could get a partial request by manually clicking the HTTP server request block less than 200 msecs after invoking the request from Squeak, but you'd need to be super fast! From playing the MicroBlocks "What's your reaction time game" I know that it is a challenge to react to a random stimulus in under 200 milliseconds.

    Why is there such a long delay between the packets sent by Squeak? I'm guessing that Squeak is not disabling the socket's "Nagle algorithm". The Nagle algorithm attempts to increase efficiency by packing more data into each TCP packet. It is intended for clients that write a few characters at a time to the socket. When just a few characters are written to the socket (e.g. "hello"), the Nagle algorithm waits a bit before sending out the data in case the client writes some more data. In this case, the Nagle algorithm isn't helpful since after writing "hello" the client is done. However, the Nagle algorithm doesn't actually send the packet containing "hello" until after a timeout. The timeout is apparently between 100 and 200 milliseconds.

    This behavior is definitely a bug in the MicroBlocks library. HTTP server request should collect an entire request before returning. (It can tell when a request is complete by checking the "Content-Length:" header.)

    Will fix.

    Meanwhile, the delay hack may help when you are working with short POST requests from Squeak.

    Abstractions are great until something fails; then you may need to look inside the abstract to figure out what when wrong.

  9. Wenjie Wu reporter

    What a great and wonderful analysis!

    my guess is that it collects data in an internal buffer. When you do a request from Squeak, then manually click the HTTP server request block, enough time has elapsed for the entire HTTP request to get buffered, even if it was transported in several TCP packets. However, when the HTTP server request block is called in the server loop, it returns as soon as the first TCP packet is received, even though the entire request has not yet been received.

    This theory totally convinced me! It explains all the symptoms very perfectly.

    but you'd need to be super fast! From playing the MicroBlocks "What's your reaction time game"

    It’s a crazy hacking game! But I won the challenge :-)

    Meanwhile, the delay hack may help when you are working with short POST requests from Squeak.

    Abstractions are great until something fails; then you may need to look inside the abstract to figure out what when wrong.

    The comprehensibility of Squeak is so good that it's great fun to explore inside, although sometimes it's hard fun. I'm going to lift the hood , get inside and follow your guide to explore the packing algorithm of WebClient.

    ‌

  11. Wenjie Wu reporter

    Got it , thank a lot !

    By the way, I'd like to share the context in which I encountered the problem.

    I've built a dotPack(NeoPixel) simulator in Squeak, and I want it to exchange data with a real dotPack driven by MicroBlocks. A simple way is to use http communication, so I want to send a pixel coordinates data to MicroBlocks, puting the data (eg: 5,6,red) in the request body. Then I encountered this problem.

    At first, I wanted to use http get to send the data http://192.168.1.7/setpixel?x=5&y=6&color=red , but the MicroBlocks http server library doesn't have the block to get url parameters.

    I think for most use case, the url parameter may be adequate. It is very lightweight and more convenient than the request body.

    Since url parameters are key-value(s) style data, it seems to be a problem to represent them as simple block in MicroBlocks. Maybe one possibility is to represent the value as a list in order.

  12. Dariusz Dorożalski

    Even POSTed data must be parsed somehow. With the power of split query string can be easily extracted from URL Querystring.png

    It can be further simplified with positional parameters (REST style)

    PathInfo.png

    .BTW, there is something wrong with the "curl" request. Properly formed HTTP POST request should have content (body) separated from headers with an empty line (CR+LF only)

  13. John Maloney repo owner

    Nice! Thanks for the suggestions. The items in your first example an be further split by "=" to get the value (item 2) once you've found the one beginning with a keyword of interest.

  15. John Maloney repo owner

    Very nice! What is the LED display you are using? Is it the LED backpack?

    I just added UDP support for ESP boards. UDP is super fast for small messages that can fit into a packet (up to ~500 bytes is guaranteed to fit into a packet on all networks). I'm not sure if Squeak has UDP socket support built in. If not, you might be able to use netcat via OSProcess.

    It looks like "curl" is working fine for your drawing application, but UDP might be a better choice for things like real-time game control. Of course, your application needs to be robust in the face of dropped packets, but when streaming controller updates continuously a few dropped packets can usually be tolerated.

  16. Wenjie Wu reporter

    Is it the LED backpack?

    yes it is.

    It works perfectly for me! I love the lightness and simplicity of the UDP library.

  17. John Maloney repo owner

    Yes, I'm enjoying using UDP. I am helping someone with a project that will show NeoPixel animations on ESP8266-based Christmas tree ornaments called "Snopixels". The Christmas tree will have its own private WiFi network. There will be up to 30 Snopixels on a tree, controlled by a central server that provides both IR and HTTP interfaces to the outside world.

    Our first design used HTTP but the HTTP calls were taking ~250 msecs and we couldn't use "keep-alive" because I'm sure that ESP8266 can't handle 30 TCP/IP sockets at once. So it became clear that HTTP simple wouldn't scale well enough. UDP, being connectionless, has much better properties for this application, including the ability to broadcast a message to all 30 Snopixels at the same time.

    We were originally planning to use an ESP8266 as the WiFi hotspot for the tree, but we discovered that the ESP8266 "softAP" hotspot can only handle four stations. Not surprising. A bit more surprising is that a Raspberry Pi acting as a hotspot also cannot handle 30 stations.

    Our current plan is to use a low-cost (~$20) WiFi router which, in theory, can handle 30+ stations as long as they don't use much bandwidth.

    This is an interesting challenge. The original proposal was to use some sort of wired network for communication. I thought WiFi would be simpler but I'm starting to see that a wired network has advantages, although it would requires more wiring. But we already need to run wires to distribute power to each Snopixel, so adding an additional twisted pair might not be difficult. I don't know much about CAN bus, but that could be a good technology this applications like this in the future.

  18. Wenjie Wu reporter

    Thank you for sharing the interesting user case and the technical decision considerations. I love the shiny stuff, I think I'll find the opportunity to build these shiny stuff with MicroBlocks at home :-)

    in theory, can handle 30+ stations as long as they don't use much bandwidth.

    In my experience handling 30 stations and keeping them online consistently does not seem to be easy.

    This is an interesting challenge. The original proposal was to use some sort of wired network for communication. I thought WiFi would be simpler but I'm starting to see that a wired network has advantages, although it would requires more wiring. But we already need to run wires to distribute power to each Snopixel, so adding an additional twisted pair might not be difficult. I don't know much about CAN bus, but that could be a good technology this applications like this in the future.

    The DMX512 is probably a good solution, It is designed to serve the shiny stage devices. It can easily handle thousands of devices with extremely high real-time performance. Thousands of DMX512 devices can be daisy chained together. When wireless network access is needed, only ArtNet (ArtNet is over UDP) service (ArtNet to DMX512) is required at the beginning of the daisy chain.

    DMX512 and ArtNet are simple and lightweight， a esp8266 board is enough to easily afford them.

    I have some experience in this area and it is quite cheap, easy and stable and reliable.

    ‌

  19. John Maloney repo owner

    Thanks for telling me about DMX512. It is much simpler than CAN it looks like the protocol could probably implemented using the standard UART built into most microcontrollers with a bit of extra code to detect the break and mark-after-break that signals the start of a new packet. I also imagine that for something like the Snopixel project one could just use a twisted pair wire and GVS connectors rather than the official cables and connectors specified in the standard.

    When you've used DMX512, were you able to implement it with a UART or did you use some sort interface breakout board?

    Given the timing, I suspect that DMX512 was influenced by the MIDI standard for electronic musical instruments that was developed around 1981. In fact, MIDI is often been used for theatrical lighting control (as well as for pyrotechnic shows). But DMX512 has a much faster data rate -- 250 kbits/second vs. MIDI's 31 kbits/sec. The DMX512 protocol also allows for 8-bit bytes whereas MIDI uses one bit for message framing (the command byte at the start of MIDI message has the bit set, data bits do not) so MIDI data bytes have only 7-bits. That makes it cumbersome to send some kinds of data, such as UTF-8 text. Of course, one can base-64 encode 8-bit data to make it fit into seven bits, but that requires adding encoding/decoding steps to your communication design.

  20. Wenjie Wu reporter

    When you've used DMX512, were you able to implement it with a UART or did you use some sort interface breakout board?

    I don't know anything about this. I would like to share my use case, in CodeLab we built a programmable space where all devices in the space can be programmed in real time and we use DMX512 is to control various lights.

    One day I want to use MicroBlocks and ESP32 to make a wearable device that correlates the dancer's movements with the lighting changes.
    At the time we were using a DMX512 controller bought from taobao to control the lights (it would convert the ArtNet messages it received into DMX512 messages)

    So I just needed to add ArtNet support to the MicroBlocks vm, which was simple, Platformio platform has a lot of open source libraries(https://github.com/hideakitai/ArtNet), and I simply wrapped them into Blocks. It works very well ！

    But now we have UDP library ! So we can build ArtNet directly in the MicroBlocks IDE.

    Because the DMX512 controller bought from taobao is expensive. I observed that when I needed better screen and user-friendly buttons, its price skyrocketed.

    So I got an idea, why don't we build it in MicroBlocks, the advantage of using MicroBlocks is that our computer display can be our hardware display/LCD screen at any time, we don't need a hardware screen and buttons to debug and set network! (I think it is a very good use case for MicroBlocks)

    I searched in Platformio for some open source projects and found that building a DMX512 controller with ESP32 was not difficult, but since I had already purchased a controller from taobao, I didn't really make one myself.

    ‌

  21. Wenjie Wu reporter

    By the way , since SqueakJS is compatible with many old Squeak images, I am reading Squeak News in my browser and happened to read a series of interviews with you, they are really really interesting！

    I'm so happy to read more about the story behind Squeak and Morphic's design ideas.

    ‌

    ‌

  22. John Maloney repo owner

    It's cool that SqueakJS let's you run all those old Smalltalk images. Super fun that you found that old interview in Squeak News. :-)

    Coincidentally, I watched the Computer History Museum "Making Smalltalk" last night. Unfortunately, the audio quality in Alan's video was bad, so it difficult to understand what he was saying (even for a native English speaker). Maybe they will make those clips available with subtitles at some point. But very interesting to hear Adele and Dan talk about the history of the Learning Research Group.

    But now we have UDP library ! So we can build ArtNet directly in the MicroBlocks IDE.

    That would be cool. I didn't look at the ArtNet protocol; is it easy to implement?

    I did some research into using UART's to implement the DMX512 on a Picaxe thread. It is possible, although a bit tricky to deal with the breaks and mark-after-break. I think if I were going to "roll my own" serial communication system I'd use normal serial and do packet framing some other way, perhaps using the xmodem approach of having a reserved frame start byte that is represented by a two-byte escape sequence when it appears in the body of a packet.

    However, after exploring both the software/protocol and hardware challenge to creating a 30+ node Christmas-tree network, the I'm thinking that using WiFi is actually pretty good and cost-effective approach. The ESP8266 nodes already have the WiFi hardware, so the only additional cost is a $20 router to tie them together.

    Also, because these trees are be on display in an outdoor park for six weeks, the fact that we don't need additional wires for the network is a plus. We still need power wires, though. The Snopixels are sealed into plastic balls with just a small hole for the power wires, and that hole gets sealed with glue. John Sze, who's been doing this project with MicroBlocks for a four or five years, says that they have had very few failures and are able to recover and reuse the Snopixel boards. (I assume they have to break open the plastic shell.)

  24. John Maloney repo owner

    We need 30 Snopixels, so ESP-NOW won't quite work. (Note: there is already optional ESP-Now support in the code, thanks to Wenjie. You just need to build and install the firmware with that option enabled.)

    UDP is working great so far, and it's useful to be able to interoperate with laptop/desktop computers, not just ESP boards. Browsers don't support UDP but I've been using Python for testing. Very easy!

    I didn't know about WiFi-Mesh. It sounds a bit like the One Laptop Per Child mesh networking system. We don't need long range for the Snopixel project but it's good to know about that feature for future projects.

  25. Wenjie Wu reporter

    Coincidentally, I watched the Computer History Museum "Making Smalltalk" last night. Unfortunately, the audio quality in Alan's video was bad, so it difficult to understand what he was saying (even for a native English speaker)

    Just like Dan said: unanticipated. And hope they will make those clips available with subtitles at some point.

    That would be cool. I didn't look at the ArtNet protocol; is it easy to implement?

    My current knowledge of it is still at the level of adding block wrappers to the open source C code. If my next project uses ArtNet, I will likely try to implement it using UDP blocks.

    However, after exploring both the software/protocol and hardware challenge to creating a 30+ node Christmas-tree network, the I'm thinking that using WiFi is actually pretty good and cost-effective approach. The ESP8266 nodes already have the WiFi hardware, so the only additional cost is a $20 router to tie them together.

    If there are no Wi-Fi dropping issue in the test, I agree this should be the best solution.

    Also, because these trees are be on display in an outdoor park for six weeks, the fact that we don't need additional wires for the network is a plus. We still need power wires, though. The Snopixels are sealed into plastic balls with just a small hole for the power wires, and that hole gets sealed with glue. John Sze, who's been doing this project with MicroBlocks for a four or five years, says that they have had very few failures and are able to recover and reuse the Snopixel boards. (I assume they have to break open the plastic shell.)

    Cool ! It's so cool to see more projects powered by MicroBlocks in the real world! Very much looking forward to seeing the shared video!

    ‌

    ‌

  27. John Maloney repo owner

    Cool! Yes, UDP is fast, which makes it great for real-time, interactive applications like this one.

  29. John Maloney repo owner

    Thanks! You're right, that's much easier to understand. During the live presentation I think they must have played Alan's videos through speakers and used a mic in the room to send the sound out to the remote viewers. In the process, it picked up a lot of reverb and room noise.

  30. Log in to comment
Assignee
–
Type
bug
Priority
major
Status
new
Votes
0
Watchers
4
Jira: the preferred issue tracker for Bitbucket. Join the team!