More fine grained timing in csv file

[Abram Hindle]

Based on this file https://pizza.cs.ualberta.ca/gm/uploads/data/agram.us_achromaticmetaphor_agram_17.agram_no_logcat.X.no_batch.20180726T0320Z.tar.gz

I look in the CSV file and I see measurements with the same time. That's probably OK but don't have a bit more decimal places for the time column?

76.976,X,118.347,4.155,0.492
76.980,X,118.347,4.155,0.492
76.980,X,118.347,4.155,0.492
76.980,X,128.203,4.151,0.532
76.980,X,113.419,4.151,0.471
76.980,X,118.347,4.155,0.492
76.980,X,118.347,4.155,0.492
76.984,X,108.491,4.155,0.451
76.984,X,108.491,4.155,0.451
76.984,X,118.347,4.155,0.492
76.984,X,113.419,4.160,0.472
76.984,X,118.347,4.160,0.492
76.988,X,118.347,4.155,0.492
76.988,X,113.419,4.160,0.472
76.988,X,118.347,4.155,0.492
76.988,X,113.419,4.155,0.471
76.988,X,108.491,4.155,0.451
76.992,X,123.275,4.155,0.512
76.992,X,113.419,4.155,0.471
76.992,X,113.419,4.155,0.471
76.992,X,123.275,4.155,0.512
76.992,X,108.491,4.160,0.451
76.992,X,103.563,4.155,0.430
76.996,X,123.275,4.155,0.512
76.996,X,113.419,4.155,0.471
76.996,X,108.491,4.155,0.451
76.996,X,123.275,4.155,0.512
76.996,X,142.988,4.131,0.591
77.000,X,152.844,4.116,0.629

[Abram Hindle]

Actually I'm a little concerned about performance

$ cat data.csv  | awk  -F, '{print $1}' | sort  | uniq -c | sort -n | awk '{print $1}'  | uniq -c

   2755 1
   2289 2
   2270 3
   2469 4
   7901 5
   5577 6
     77 7
      1 10
      2 15
      4 16
      3 17
      1 18
      1 19
      9 20
     11 21
     17 22
     23 23
     38 24
     51 25
    123 26
    202 27
    384 28
    625 29
   1606 30
    892 31
      2 32

So lots of 4 millisecond stretches have only 1 measurement And some have up 31 measurements.

After about 4 seconds the measurement rate goes down to 1 per millisecond and stays there.

$ cat data.csv  | awk  -F, '{print $1}' | sort -n  | uniq -c
      1 time
     10 0.001
     16 0.002
     15 0.003
     21 0.004
     26 0.005
     26 0.006
     26 0.007
     26 0.008
     26 0.009
     ...
          1 7.081
      5 7.082
      5 7.086
      5 7.090
      6 7.094
      5 7.098
      6 7.102
      5 7.106
      5 7.110
      1 7.111
      2 7.114
      3 7.115

So there's a problem with reading that it slows down for some reason. Maybe the python is slow, maybe the arduino.

[greenminer_kyudai]

We thought that the cause of the large number of reads in the first few seconds in python's reading is that the data is stored in the buffer first

[greenminer_kyudai]

So we inserted flush_buffer() but it didn't affect

[Abram Hindle]

OK so you should write a test case to measure median (not mean) measurements per second.

This will allow you to debug easier.

I think the python script might be slow as well.

Test script should be:

run test on phone measure energy get data.csv

hindle1@frail:/tmp$ cat data.csv | awk -F, '{print $1}'  | sed -e 's/\.[0-9]*$//' | uniq -c 
      1 time
  28913 0
  29483 1
  29401 2
  29333 3
   1335 4
   1335 5
   1335 6
   1340 7
   1335 8
   1335 9
   1335 10
   1335 11
   1335 12
   1338 13
   1337 14
   1335 15
   1335 16
   1335 17
   1335 18
   1335 19
   1340 20
   1335 21
   1335 22
   1335 23
   1335 24
   1335 25
   1341 26
   1334 27
   1336 28
   1334 29
   1335 30
   1335 31
   1338 32
   1338 33
   1334 34
   1336 35
   1334 36
   1335 37
   1335 38
   1340 39
   1336 40
   1334 41
   1336 42
   1334 43
   1336 44
   1336 45
   1338 46
   1336 47
   1334 48
   1336 49
   1334 50
   1336 51
   1340 52
   1335 53
   1335 54
   1334 55
   1336 56
   1334 57
   1339 58
   1337 59
   1335 60
   1335 61
   1334 62
   1336 63
   1334 64
   1341 65
   1335 66
   1335 67
   1335 68
   1335 69
   1335 70
   1337 71
   1338 72
   1335 73
   1335 74
   1335 75
   1335 76
      1 77

Get a median of the first column and you can see how many measurements per second. It looks like 1.3k measurements. I suspect maybe you have a python issue.

[greenminer_kyudai]

We modify flush_buffer() in arduino.py and baud rate from 57200 to 115200.

[greenminer_kyudai]

cat data.csv | awk -F, '{print $1}'  | sed -e 's/\.[0-9]*$//' | uniq -c
   1 time
2665 0
2670 1
2670 2
2671 3
2680 4
2670 5
2669 6
2670 7
2670 8
2670 9
2675 10
2676 11
2670 12
2670 13
2670 14
2670 15
2670 16
2677 17
2674 18
2669 19
2670 20
2670 21
2670 22
2672 23
2678 24
2671 25
2670 26
2670 27
2670 28
2670 29
2674 30
2676 31
2670 32
2670 33
2670 34
2670 35
2671 36
2678 37
2672 38
2670 39
2670 40
2670 41
2670 42
2671 43
2679 44
2670 45
2670 46
2670 47
2670 48
2670 49
2676 50
2675 51
2670 52
2670 53
2670 54
2669 55
2672 56
2679 57
2670 58
2670 59
2670 60
2670 61
2670 62
2674 63
2677 64
2670 65
2670 66
2670 67
2669 68
2670 69
2678 70
2673 71
2670 72
2670 73
2670 74
2670 75
2671 76
  10 77

[greenminer_kyudai]

and modify time column in data.csv

time,id,mA,V,W
0.00076,X,251.407,4.102,1.031
0.00089,X,271.120,4.102,1.112
0.00095,X,251.407,4.102,1.031
0.00101,X,236.623,4.102,0.971
0.00106,X,246.479,4.107,1.012
0.00112,X,256.335,4.102,1.051
0.00117,X,271.120,4.102,1.112
0.00122,X,251.407,4.116,1.035
0.00128,X,241.551,4.107,0.992
0.00134,X,187.341,4.111,0.770
0.00145,X,246.479,4.107,1.012
0.00151,X,246.479,4.111,1.013
0.00162,X,246.479,4.102,1.011
0.00172,X,251.407,4.107,1.033
0.00178,X,231.694,4.107,0.952
0.00183,X,271.120,4.107,1.113
0.00190,X,271.120,4.107,1.113
0.00195,X,266.192,4.107,1.093
...

[Abram Hindle]

Great.

[greenminer_kyudai]

increase more decimal places and stabilize read rate