Wiki
Clone wikiscl-manips-v2 / docu / scl-distrib
Distributed Interfaces for SCL
Requirements for the Multi-Robot Interface
add stuff here...
Requirements for the Web Interface
Motivation
SCL's focus on providing a strong back-end framework to tie together different simulation and control modules has limited its utility for many common use cases. In particular, there is a strong need for:
- Data introspection
- Present options : C++ logging and offline rendering || tedious & complex OpenGL/Qt programming
- Desired : Real-time introspection through a command interface && high level interaction using some scripting language
- Systematic logging
- Present options : Hard coded C++ logging; lots of crufty code and very tedious to look at everything
- Desired : Simple logging with dynamic log introspection, rendering and adjustable sampling
Given the requirements, it was almost imperative to interface some sort of scripting language with SCL. The obvious choices were either python or javascript. In the end, many developments in pseudo real-time rendering and webGL made javascript the winner. The biggest advantage is being able to push complex simulations to one's website rapidly, and be completely platform independent.
SCL as a Distributed System
The program to enable web interfaces introduces a strong and systematic push for SCL to become a distributed system. This requires many systemic changes in how things are structured and promises to have strong repercussions for SCL's future.
Summarizing our design choices:
- Use JSON as a serialization and data sharing format
- Readable, easily parsed, and is also YAML.
- Can be converted
- Store SCL data structures as bulk JSON in Redis, 2 keys per robot (static, dynamic)
- Node server on the backend
- Javascript (or coffeescript) on both frontend and backend keeps things simple
- Great Redis bindings with node_redis or even redis-atomic-json.
- Socket.IO for streaming data from server to client
- Fast, uses websockets
- Super simple with Node
Some numbers
Fast Redis IO with Raw Data
- Stream a Puma's joint angles (a string of doubles) and energy using hiredis (sync) while running the scl integrator..
Iterations : 50,000 Time : 5.65 sec
- Raw SCL integration time:
Time : 4.39 sec
Real-time Redis IO with 4ms Integ and 25Hz JSON SRobotIO Flush
- Stream a Puma's IO state and energy using hiredis (sync) while running the scl integrator (@250Hz)..
Total iterations : 7,409 Total real-world time : 29.6502 Total simulated time : 29.636
- Note that we don't need a raw SCL comparison because there is zero lag (and the sim is sleeping most of the time)... See ecd0f5ecb440be535ef1dd46f70755f6c3250f5a for a sample app...
Fast Redis IO with 4ms Integ and 1:1 JSON SRobotIO Flush
- Looks like we can dump a Puma's state into redis at about 5KHz
Total iterations : 50,000 Total real-world time : 10.0302 Total simulated time : 200
- Removing the redis calls from the sim, we get (raw integration time)
Total iterations : 50,000 Total real-world time : 4.39417 Total simulated time : 200
- Keeping all the serialization and message formatting etc. Just avoid the actual redis call
Total iterations : 50,000 Total real-world time : 7.19048 Total simulated time : 200
- See ecd0f5ecb440be535ef1dd46f70755f6c3250f5a for a sample app...
- Can probe values in redis this way...
127.0.0.1:6379> keys SCL:PumaBot* 1) "SCL:PumaBot:Graphics" 2) "SCL:PumaBot:Parsed" 3) "SCL:PumaBot:Energy" 4) "SCL:PumaBot:IO" 127.0.0.1:6379> get SCL:PumaBot:Energy "395.235732" 127.0.0.1:6379> get SCL:PumaBot:IO "{\"actuators_\":{\"force_gc_commanded_\":[0,0,0,0,0,0]},\"dof_\":6,\"sensors_\":{\"ddq_\":[-1.833860,2.812860,6.156520,-0.1120020,0.09710940,0.002315630],\"dq_\":[-0.3392980,3.249640,1.913830,0.2955420,4.324490,-1.623790000000000e-05],\"force_gc_measured_\":[0,0,0,0,0,0],\"forces_external_\":null,\"q_\":[-0.1211070,1.054540,99.76580,-5.329780,114.6510,0.003457050]}}\n"
Hayk's speed notes
I'm testing scl_redis_serv_puma with no sleeping. Running 50k iterations, counting the loop time by dividing the real-world time by 50000. Compiled in release mode but steps taken to make sure steps don't get optimized out.
Starting out with no integration even, with a completely blank simulation loop except for energy printout every 1000 steps.
Do basically nothing: 0.58 us/step
This will form the baseline to test each part by itself. Below is each component by itself in the loop, to let us know what kind of bottleneck it causes. Keep in mind that for a 10 kHz loop, we need 100 us/step (including everything).
Send short dummy command w/ Redox sync: 37 us/step
Send SCL:PumaBot:IO string (no serialization) w/ Redox sync: 42 us/step
Send short asynchronous command w/ Redox async: 4.2 us/step
Send SCL:PumaBot:IO string (no serialization) w/ Redox async: 9.2 us/step
Call serializeToJson, don't write: 39-68 us/step (quite variable)
Call serializeToJson, and writer.write: 52-86 us/step (quite variable)
serializeToJson, writer.write, and Redox async: 61-96 us/step
Just integration, nothing else: 126-186 us/step
Integration, serialization, Redox async: 234-360 us/step
Conclusions: Sending an asynchronous command is very fast, and even with no compression (full string output) takes under ten microseconds. The JSON serialization takes 6-9x longer than sending commands, with about 70/30 split in time between serializeToJson and writer.write. For some reason, my integration is taking quite a long time (cannot reach 10kHz even by itself, which is odd because in my iron dome project I could do 10kHz).
I tried just using a stringstream and manually creating the JSON string. Performance was very similar to just using the JSON serialization, which probably means that json-cpp uses stringstreams. Also, a note is that the sensor and actuator data should be split up in practice - a controller should only be outputting actuator data, and a driver should only be outputting sensor data.
All said, we aren't hitting 10kHz right now on my laptop, but rather 3-4 kHz in real time. However, that's with strangely large integration times. We have room to try to optimize, but for now I'll move to the web side of things.
Update: I realized this example was using a different integrator than I had been before. I tried out Tao instead of Spatial (not sure what the tradeoffs are), and it is much faster. I can get something like 8kHz. However, energy seems to be all over the place.
Technical Issues and Challenges
Issue 1 : Real-time introspection
- SCL should be completely reentrant ASAP!
- All SCL data structures should support serialization
Updated