Excessive CPU usage and stuck data upload

Issue #293 resolved
Daniel Jagszent
created an issue

S3QL: 2.26
Dugong: 3.7.1
LLfuse: 1.3.2
Backend: Swiftks (OVH)

After some time of usage the S3QL filesystem is in this state (Maximum cache size is 100 GiB):

$ s3qlstat /cloud/storage1/
Directory entries:    5204
Inodes:               5206
Data blocks:          5956
Total data size:      4.16 TB
After de-duplication: 4.16 TB (100.00% of total)
After compression:    4.16 TB (100.01% of total, 100.01% of de-duplicated)
Database size:        3.13 MiB (uncompressed)
Cache size:           99.6 GiB, 173 entries
Cache size (dirty):   2.80 GiB, 1 entries
Queued object removals: 0

This one dirty cache entry never gets uploaded. S3QL uses 100% CPU. Here is a small part of the debug output of mount.log:

2018-03-18 17:02:19.974 1692:Metadata-Upload-Thread s3ql.metadata.upload_metadata: Cycling metadata backups...
2018-03-18 17:02:19.974 1692:Metadata-Upload-Thread s3ql.metadata.cycle_metadata: Backing up old metadata...
2018-03-18 21:02:47.387 1692:Thread-13 s3ql.backends.swift._do_authentication_expired: OpenStack auth token seems to have expired, requesting new one.
2018-03-18 21:04:54.555 1692:Thread-3 s3ql.backends.swift._do_authentication_expired: OpenStack auth token seems to have expired, requesting new one.
2018-03-18 21:05:52.678 1692:Thread-18 s3ql.backends.swift._do_authentication_expired: OpenStack auth token seems to have expired, requesting new one.
2018-03-18 21:11:37.912 1692:Thread-6 s3ql.backends.swift._do_authentication_expired: OpenStack auth token seems to have expired, requesting new one.
2018-03-18 21:15:22.165 1692:Thread-19 s3ql.backends.swift._do_authentication_expired: OpenStack auth token seems to have expired, requesting new one.
2018-03-18 21:25:32.173 1692:Thread-4 s3ql.backends.swift._do_authentication_expired: OpenStack auth token seems to have expired, requesting new one.
2018-03-18 21:25:42.956 1692:Thread-5 s3ql.backends.swift._do_authentication_expired: OpenStack auth token seems to have expired, requesting new one.
2018-03-19 03:46:30.464 1692:Thread-8 s3ql.backends.swift._do_authentication_expired: OpenStack auth token seems to have expired, requesting new one.
2018-03-19 12:13:00.350 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.351 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12789, blockno=0>
2018-03-19 12:13:00.352 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.352 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12793, blockno=0>
2018-03-19 12:13:00.354 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.354 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.354 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12797, blockno=0>
2018-03-19 12:13:00.358 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.358 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.359 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12795, blockno=0>
2018-03-19 12:13:00.359 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.359 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.359 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12796, blockno=0>
2018-03-19 12:13:00.359 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.360 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.360 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12794, blockno=0>
2018-03-19 12:13:00.360 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.360 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.360 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12792, blockno=0>
2018-03-19 12:13:00.361 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.361 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.361 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12803, blockno=0>
2018-03-19 12:13:00.361 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.362 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.362 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12799, blockno=0>
2018-03-19 12:13:00.362 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.362 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.363 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12801, blockno=0>
2018-03-19 12:13:00.363 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.363 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.363 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12802, blockno=0>
2018-03-19 12:13:00.363 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.364 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.364 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12800, blockno=0>
2018-03-19 12:13:00.364 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.365 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.365 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12804, blockno=0>
2018-03-19 12:13:00.365 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.365 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.365 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12809, blockno=0>
2018-03-19 12:13:00.366 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.366 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12807, blockno=0>
2018-03-19 12:13:00.366 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12810, blockno=0>
2018-03-19 12:13:00.366 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.366 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.367 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12806, blockno=0>
2018-03-19 12:13:00.367 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.367 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.367 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12812, blockno=0>
2018-03-19 12:13:00.367 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.368 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.368 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12811, blockno=0>
2018-03-19 12:13:00.369 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.369 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.370 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12813, blockno=0>
2018-03-19 12:13:00.370 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.370 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.370 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12814, blockno=0>
2018-03-19 12:13:00.371 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.371 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.371 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12798, blockno=0>
2018-03-19 12:13:00.372 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.372 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.372 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12816, blockno=0>
2018-03-19 12:13:00.372 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.372 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.372 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12815, blockno=0>
2018-03-19 12:13:00.373 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.373 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.373 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12805, blockno=0>
2018-03-19 12:13:00.373 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.374 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.374 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12817, blockno=0>
2018-03-19 12:13:00.374 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.374 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.374 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12772, blockno=0>
2018-03-19 12:13:00.375 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.375 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.375 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12805, blockno=1>
2018-03-19 12:13:00.375 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.376 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.376 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12819, blockno=0>
2018-03-19 12:13:00.376 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.376 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.376 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12818, blockno=0>
2018-03-19 12:13:00.377 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.377 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.377 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12770, blockno=0>
2018-03-19 12:13:00.378 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12822, blockno=0>
2018-03-19 12:13:00.378 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.378 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.378 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12823, blockno=0>
2018-03-19 12:13:00.378 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.378 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.380 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12820, blockno=0>
2018-03-19 12:13:00.380 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.380 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.380 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12821, blockno=0>
2018-03-19 12:13:00.381 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.381 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.381 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.381 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.381 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12805, blockno=2>
2018-03-19 12:13:00.381 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.382 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.382 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12824, blockno=0>
2018-03-19 12:13:00.382 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.382 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.382 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12772, blockno=1>
2018-03-19 12:13:00.383 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.383 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.383 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12808, blockno=0>
2018-03-19 12:13:00.384 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.384 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.384 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12772, blockno=2>
2018-03-19 12:13:00.384 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.385 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.385 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12772, blockno=3>
2018-03-19 12:13:00.385 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.385 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.385 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12825, blockno=0>
2018-03-19 12:13:00.386 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.386 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.386 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12828, blockno=0>
2018-03-19 12:13:00.386 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.387 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.387 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12829, blockno=0>
2018-03-19 12:13:00.387 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.387 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.388 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12826, blockno=0>
2018-03-19 12:13:00.388 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.388 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.388 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12830, blockno=0>
2018-03-19 12:13:00.389 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.389 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.389 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12827, blockno=0>
2018-03-19 12:13:00.390 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.390 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.390 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12832, blockno=0>
2018-03-19 12:13:00.390 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.391 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.391 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12831, blockno=0>
2018-03-19 12:13:00.391 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12835, blockno=0>
2018-03-19 12:13:00.392 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.392 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.392 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12837, blockno=0>
2018-03-19 12:13:00.392 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.392 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.392 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12838, blockno=0>
2018-03-19 12:13:00.393 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.393 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.393 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12840, blockno=0>
2018-03-19 12:13:00.393 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.394 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.394 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12839, blockno=0>
2018-03-19 12:13:00.395 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.395 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.395 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12836, blockno=0>
2018-03-19 12:13:00.396 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.396 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.397 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12834, blockno=0>
2018-03-19 12:13:00.397 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.397 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.397 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12842, blockno=0>
2018-03-19 12:13:00.398 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.398 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.398 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12845, blockno=0>
2018-03-19 12:13:00.398 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.398 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.399 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12843, blockno=0>
2018-03-19 12:13:00.399 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.399 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.399 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12841, blockno=0>
2018-03-19 12:13:00.399 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.400 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.400 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12849, blockno=0>
2018-03-19 12:13:00.400 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.400 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.400 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12844, blockno=0>
2018-03-19 12:13:00.401 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.401 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.401 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.401 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.401 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12848, blockno=0>
2018-03-19 12:13:00.401 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.402 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.402 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12847, blockno=0>
2018-03-19 12:13:00.402 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.403 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.403 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.403 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.403 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12850, blockno=0>
2018-03-19 12:13:00.403 1692:Thread-8 dugong.co_write: done
2018-03-19 12:13:00.404 1692:Thread-6 dugong.co_write: done
2018-03-19 12:13:00.404 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12852, blockno=0>
2018-03-19 12:13:00.404 1692:Thread-8 dugong.eval_coroutine: polling
2018-03-19 12:13:00.405 1692:Thread-6 dugong.eval_coroutine: polling
2018-03-19 12:13:00.405 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12851, blockno=0>
2018-03-19 12:13:00.405 1692:Thread-8 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.405 1692:Thread-6 dugong.co_write: start (len=65536)
2018-03-19 12:13:00.405 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12855, blockno=0>
2018-03-19 12:13:00.406 1692:Thread-8 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.406 1692:Thread-6 dugong._co_send: trying to send 65536 bytes
2018-03-19 12:13:00.406 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12857, blockno=0>
2018-03-19 12:13:00.406 1692:Thread-6 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.407 1692:Thread-8 dugong._co_send: sent 65536 bytes
2018-03-19 12:13:00.407 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12856, blockno=0>
2018-03-19 12:13:00.407 1692:Thread-6 dugong._co_send: done
2018-03-19 12:13:00.407 1692:Thread-8 dugong._co_send: done
2018-03-19 12:13:00.407 1692:CommitThread s3ql.block_cache.upload_if_dirty: started with <CacheEntry, inode=12853, blockno=0>
^C

Comments (11)

  1. Nikolaus Rath repo owner

    Thanks for the report! Just to make sure: can you rule out that this one entry isn't simply being modified over and over again while being uploaded? The log doesn't contain any errors... but it is also just for one upload attempt.

    Out of curiosity: why aren't you using a smaller block size? Anything above, say, 500 MB seems rather pointless to me.

  2. Daniel Jagszent reporter

    This S3QL filesystem is for backup purposes and only accessed by one program (Bareos storage daemon). Files are only appended and never read (only for restores). Backups start in the evening and are done well before noon (were this debug log is from). So I doubt that this block gets accessed/changed over and over at this time.

    OVH's Swift storage is normally performant enough (~300-500MBit/s per connection) but a small percentage of uploads somehow get stuck with upload rates of as low as 20KB/s. That's maybe why this dirty cache block is left. (Not a S3QL specific problem - other clients like rclone also experience this). This would normally not be a problem since it eventually gets uploaded successfully and I do not care if it takes days to do so.

    But why is S3QL using 100% CPU in the CommitThread in this case? Looks like some kind of aggressive polling at an edge case.

    Out of curiosity: why aren't you using a smaller block size? Anything above, say, 500 MB seems rather pointless to me.

    I will probably lower the block size with the next filesystem to around 200MB. I wanted to optimize for a small S3QL database size and few objects in the Swift container. I have/had other S3QL filesystems on OVHs Swift storage and they get slower with millions of objects in the container and a small S3QL metadata database is also good. But a max block size of 3GB was probably going over the top.

  3. Nikolaus Rath repo owner

    Yeah, this does sound like a bug then. I strongly suspect an edge case when there is only a single object in cache. As an experiment, could you try to create a few more objects in the cache (just a few MB each) and see if that gets un-stucks upload of the remaining large object?

  4. Nikolaus Rath repo owner

    As long as the object is being uploaded, I can't fathom what the CPU cycles could possibly be spent on. The upload thread should be blocked on a select() on the socket, and every other thread that deals with this object should be blocked on a mutex.

    Could you also try to get a few stacktraces to see what the different threads are doing?

  5. Daniel Jagszent reporter

    The filesystem is currently in this state again. Adding some more blocks in this state does not un-stuck the the other block / end the excessive CPU usage.

    I tried sending USR1 to S3QL but it strangely does not print stacktraces. (Nothing happens, no output in mount.log and/or in SystemD Journal)

  6. Daniel Jagszent reporter

    Here is some debugging info:

    Thread list of mount.s3ql:

    ps -T -p 1877
      PID  SPID TTY          TIME CMD
     1877  1877 ?        00:00:00 mount.s3ql
     1877  1884 ?        00:09:33 mount.s3ql
     1877  1885 ?        00:11:20 mount.s3ql
     1877  1886 ?        00:08:34 mount.s3ql
     1877  1887 ?        00:48:53 mount.s3ql
     1877  1888 ?        00:10:30 mount.s3ql
     1877  1889 ?        00:10:53 mount.s3ql
     1877  1890 ?        00:14:07 mount.s3ql
     1877  1891 ?        00:10:52 mount.s3ql
     1877  1892 ?        00:00:00 mount.s3ql
     1877  1893 ?        00:00:00 mount.s3ql
     1877  1894 ?        00:00:00 mount.s3ql
     1877  1895 ?        00:00:00 mount.s3ql
     1877  1896 ?        00:00:00 mount.s3ql
     1877  1897 ?        00:00:00 mount.s3ql
     1877  1898 ?        00:00:00 mount.s3ql
     1877  1899 ?        00:00:00 mount.s3ql
     1877  1900 ?        00:00:00 mount.s3ql
     1877  1901 ?        00:00:00 mount.s3ql
     1877  1902 ?        00:00:01 mount.s3ql
     1877  1903 ?        2-03:15:45 mount.s3ql
     1877  1904 ?        00:00:00 mount.s3ql
     1877  1906 ?        00:00:28 mount.s3ql
     1877  1907 ?        00:00:25 mount.s3ql
     1877  1908 ?        00:00:27 mount.s3ql
     1877  1909 ?        00:00:26 mount.s3ql
     1877  1910 ?        00:00:25 mount.s3ql
     1877  1911 ?        00:00:25 mount.s3ql
     1877  1912 ?        00:00:25 mount.s3ql
     1877  1913 ?        00:00:26 mount.s3ql
     1877  1914 ?        00:00:27 mount.s3ql
     1877  1915 ?        00:00:27 mount.s3ql
     1877  1916 ?        00:00:30 mount.s3ql
     1877  1917 ?        00:00:25 mount.s3ql
     1877  1918 ?        00:00:27 mount.s3ql
     1877  1919 ?        00:00:25 mount.s3ql
     1877  1920 ?        00:00:26 mount.s3ql
     1877  1921 ?        00:00:27 mount.s3ql
     1877  1922 ?        00:00:26 mount.s3ql
     1877  1923 ?        00:00:26 mount.s3ql
     1877  1924 ?        00:00:27 mount.s3ql
     1877  1925 ?        00:00:24 mount.s3ql
     1877  1926 ?        00:00:25 mount.s3ql
     1877  1927 ?        00:00:26 mount.s3ql
     1877  1928 ?        00:00:27 mount.s3ql
     1877  1929 ?        00:00:26 mount.s3ql
     1877  1930 ?        00:00:28 mount.s3ql
     1877  1931 ?        00:00:25 mount.s3ql
     1877  1932 ?        00:00:27 mount.s3ql
     1877  1933 ?        00:00:26 mount.s3ql
     1877  1934 ?        00:00:26 mount.s3ql
     1877  1935 ?        00:00:26 mount.s3ql
    

    So only one thread (1903) is the culprit.

    (gdb) attach 1903
    Attaching to process 1903
    Reading symbols from /usr/bin/python3.5...Reading symbols from /usr/lib/debug/.build-id/59/a8ef36ca241df24686952480966d7bc0d7c6ea.debug...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug/.build-id/ce/17e023542265fc11d9bc8f534bb4f070493d30.debug...done.
    done.
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.23.so...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.23.so...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libutil.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libutil-2.23.so...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libexpat.so.1...(no debugging symbols found)...done.
    Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
    Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.23.so...done.
    done.
    Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.23.so...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_compat-2.23.so...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnsl-2.23.so...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_nis-2.23.so...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.23.so...done.
    done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/_bz2.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/f9/d99c611db90cc175e50fcc2e0584133a5974fa.debug...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libbz2.so.1.0...(no debugging symbols found)...done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/_lzma.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/2b/2494a591bb8dbcc4b2a843655e602dd51ca998.debug...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols found)...done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/_hashlib.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/46/4f2c1c666276af77669a4852a3790995fd05b6.debug...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
    Reading symbols from /usr/lib/python3/dist-packages/llfuse.cpython-35m-x86_64-linux-gnu.so...(no debugging symbols found)...done.
    Reading symbols from /lib/x86_64-linux-gnu/libfuse.so.2...(no debugging symbols found)...done.
    Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.23.so...done.
    done.
    Reading symbols from /usr/lib/s3ql/s3ql/deltadump.cpython-35m-x86_64-linux-gnu.so...(no debugging symbols found)...done.
    Reading symbols from /usr/lib/x86_64-linux-gnu/libsqlite3.so.0...(no debugging symbols found)...done.
    Reading symbols from /usr/lib/python3/dist-packages/apsw.cpython-35m-x86_64-linux-gnu.so...(no debugging symbols found)...done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/_csv.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/9d/392bad0555f5b45ad26e0dc0a5842f78f7c49b.debug...done.
    done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/_opcode.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/08/299902d590d544590c361db9c4d65d58a2d21d.debug...done.
    done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/bf/9d7ce7f341d682312cebb9250b6577619d3e2a.debug...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/_multiprocessing.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/ac/8c85fffe5d1a950f01768aa861093d989f290a.debug...done.
    done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/termios.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/81/93293ce0184c1f480342cd3ce089a88bb46c04.debug...done.
    done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/22/c712b4da38f503200be2815bfc8f75157136ba.debug...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libuuid.so.1...(no debugging symbols found)...done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/_json.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/fa/3430fc8d5ab56bf93d7c8ffdf012851942eac3.debug...done.
    done.
    Reading symbols from /usr/lib/python3.5/lib-dynload/resource.cpython-35m-x86_64-linux-gnu.so...Reading symbols from /usr/lib/debug/.build-id/0e/7a57211a7a7a9e74baa6a3470d31be5743504b.debug...done.
    done.
    Reading symbols from /usr/lib/python3/dist-packages/systemd/_daemon.cpython-35m-x86_64-linux-gnu.so...(no debugging symbols found)...done.
    Reading symbols from /lib/x86_64-linux-gnu/libsystemd.so.0...(no debugging symbols found)...done.
    Reading symbols from /lib/x86_64-linux-gnu/libselinux.so.1...(no debugging symbols found)...done.
    Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.20...(no debugging symbols found)...done.
    Reading symbols from /lib/x86_64-linux-gnu/libpcre.so.3...(no debugging symbols found)...done.
    Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done.
    Reading symbols from /usr/lib/python3/dist-packages/Crypto/Cipher/_AES.cpython-35m-x86_64-linux-gnu.so...(no debugging symbols found)...done.
    Reading symbols from /usr/lib/python3/dist-packages/Crypto/Util/_counter.cpython-35m-x86_64-linux-gnu.so...(no debugging symbols found)...done.
    Reading symbols from /lib/x86_64-linux-gnu/libnss_dns.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_dns-2.23.so...done.
    done.
    Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libresolv-2.23.so...done.
    done.
    _PyObject_GenericGetAttrWithDict () at ../Objects/object.c:1057
    1057    ../Objects/object.c: No such file or directory.
    
    (gdb) continue
    ^C
    Program received signal SIGINT, Interrupt.
    (gdb) py-bt
    Traceback (most recent call first):
      File "/usr/lib/s3ql/s3ql/mount.py", line 752, in run
        if self.block_cache.upload_if_dirty(el):
      File "/usr/lib/s3ql/s3ql/mount.py", line 64, in run_with_except_hook
        run_old(*args, **kw)
      File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
        self.run()
      File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
        self._bootstrap_inner()
    (gdb) py-list
     747                    for el in list(self.block_cache.cache.values()):
     748                        if self.stop_event.is_set():
     749                            break
     750                        if stamp - el.last_access < 10:
     751                            break
    >752                        if self.block_cache.upload_if_dirty(el):
     753                            did_sth = True
     754    
     755                if not did_sth:
     756                    self.stop_event.wait(5)
     757   
    (gdb) py-locals
    self = <CommitThread(block_cache=<BlockCache(fs=<Operations(failsafe=False, fs_size=None, cache=<...>, upload_event=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7fdfedb063f0>, release=<built-in method release of _thread.lock object at remote 0x7fdfedb063f0>, acquire=<built-in method acquire of _thread.lock object at remote 0x7fdfedb063f0>, _waiters=<collections.deque at remote 0x7fdfedb03d48>) at remote 0x7fdfec86a6d8>, _flag=False) at remote 0x7fdfec86ab70>, broken_blocks={}, max_obj_size=3221225472, open_inodes={1: 1, 4: 6, 14471: 1, 14491: 1, 14463: 1}, db=<Connection(file='/root/.s3ql/swiftks:=2F=2Fauth.cloud.ovh.net=2FSBG1:bareos1=2Fstorage1.db', conn=<apsw.Connection at remote 0x7fdfedae5310>) at remote 0x7fdfec855ef0>, inodes=<InodeCache(generation=0, attrs={1: <_Inode at remote 0x7fdfec3fdb88>, 2: <_Inode at remote 0x7fdfec3fdee8>, 3: <_Inode at remote 0x7fdfec400ca8>, 4: <_Inode at remote 0x7fdfedb07168>, 14396: <_Inode at remote 0x7fdfec3fd4c8>, 14397: <_Inode at remote 0x7fdfec3fcd38>, 14398: <...(truncated)
    did_sth = False
    stamp = <float at remote 0x7fdfe83c6150>
    el = <CacheEntry at remote 0x7fdfec3f5e80>
    

    strace says this about this thread:

    strace: Process 1903 attached
    strace: [ Process PID=1903 runs in x32 mode. ]
    strace: [ Process PID=1903 runs in 64 bit mode. ]
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794752963, {1523475801, 372991000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794752969, {1523475801, 373542000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794752973, {1523475801, 373931000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794752975, {1523475801, 374263000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794752978, {1523475801, 374566000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794752981, {1523475801, 374923000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794752986, {1523475801, 375554000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794752989, {1523475801, 375680000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794752993, {1523475801, 376128000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753000, {1523475801, 377547000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753007, {1523475801, 378068000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753019, {1523475801, 379381000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753021, {1523475801, 379381000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753028, {1523475801, 380412000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753032, {1523475801, 380412000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753041, {1523475801, 382161000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753049, {1523475801, 383250000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753061, {1523475801, 385196000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753065, {1523475801, 385456000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753090, {1523475801, 388899000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753093, {1523475801, 389188000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753095, {1523475801, 389561000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753099, {1523475801, 390684000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753106, {1523475801, 391772000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753116, {1523475801, 392869000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753119, {1523475801, 392869000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753125, {1523475801, 394065000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753127, {1523475801, 394065000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753129, {1523475801, 394065000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753134, {1523475801, 394643000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753137, {1523475801, 394643000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753144, {1523475801, 396141000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753145, {1523475801, 396141000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753147, {1523475801, 396141000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753158, {1523475801, 398242000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753159, {1523475801, 398242000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753164, {1523475801, 399727000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753165, {1523475801, 399727000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753173, {1523475801, 400785000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753177, {1523475801, 400902000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753181, {1523475801, 400902000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753185, {1523475801, 401825000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753190, {1523475801, 401953000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753194, {1523475801, 402451000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753204, {1523475801, 403959000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753207, {1523475801, 403959000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753218, {1523475801, 405925000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753219, {1523475801, 405925000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753224, {1523475801, 406930000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753230, {1523475801, 407816000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753232, {1523475801, 407816000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753233, {1523475801, 407816000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753238, {1523475801, 409085000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753239, {1523475801, 409085000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753244, {1523475801, 410312000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753245, {1523475801, 410312000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753249, {1523475801, 410867000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753261, {1523475801, 413138000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753273, {1523475801, 415219000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753293, {1523475801, 418353000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753304, {1523475801, 419611000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753305, {1523475801, 419611000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753331, {1523475801, 423399000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753340, {1523475801, 424720000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753366, {1523475801, 428763000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753381, {1523475801, 430988000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753385, {1523475801, 431486000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753399, {1523475801, 433809000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753401, {1523475801, 433809000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753426, {1523475801, 438387000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753437, {1523475801, 439983000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753449, {1523475801, 441702000}, ffffffff) = 0
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753461, {1523475801, 443758000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753463, {1523475801, 444064000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50260, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753465, {1523475801, 444589000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753485, {1523475801, 448343000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753509, {1523475801, 452806000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50220, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 1
    futex(0xa50264, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 794753549, {1523475801, 459826000}, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0xa50220, FUTEX_WAKE_PRIVATE, 1)  = 0
    futex(0xa50264, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa50260, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
    ^C
    

    The CommitThread is in an endless busy loop and the only syscalls it is doing are futex calls (from which many fail with EAGAIN but that is probably ok.

  7. Log in to comment