how to debug failing features

[Vladimir Orany]

I have a problem when I fill a SQS queue with messages I can't no longer get any messages from any queue. I can't see anything going wrong in the logs. Is there any place I can inspect for details why is the SQS mock failing to reply. I got plenty of timeouts when trying to read the messages.

[Waldemar Hummer]

Enabling the DEBUG=1 environment variable should give you some debugging information. Depending on whether you use Docker or not, use either:

DEBUG=1 make docker-run

or:

DEBUG=1 make infra

If you are using the AWS CLI, can you please post the exact sequence of aws sqs ... commands you are executing? That would make it easier to reproduce the issue. Thanks

[Vladimir Orany]

I'm not quite sure yet what is the sequence that's what I'm trying to debug. I'll try to use your hint and if there is something I can share I'll come back to you.

[Vladimir Orany]

I'm not sure if this does any help but I got following stacktrace from one of my attempts

----------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 596, in process_request_thread
172.17.0.1 - - [10/May/2017 14:26:40] "POST http://sqs.eu-west-1.amazonaws.com/123456789012/vlad_manager_ManagerDailyEmail HTTP/1.1" 200 -
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
self.RequestHandlerClass(request, client_address, self)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 596, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 654, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 713, in finish
self.wfile.close()
File "/usr/lib/python2.7/socket.py", line 283, in close
File "/usr/lib/python2.7/SocketServer.py", line 654, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 713, in finish
self.wfile.close()
File "/usr/lib/python2.7/socket.py", line 283, in close
172.17.0.1 - - [10/May/2017 14:26:40] "POST http://sqs.eu-west-1.amazonaws.com/123456789012/vlad_facebook_FacebookItemsApprove HTTP/1.1" 200 -
self.flush()
File "/usr/lib/python2.7/socket.py", line 307, in flush
self.flush()
File "/usr/lib/python2.7/socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 596, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 654, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 713, in finish
self.wfile.close()
File "/usr/lib/python2.7/socket.py", line 283, in close
self.flush()
File "/usr/lib/python2.7/socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
172.17.0.1 - - [10/May/2017 14:26:41] "POST http://sqs.eu-west-1.amazonaws.com/123456789012/vlad_facebook_FacebookResponseTime HTTP/1.1" 200 -
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 596, in process_request_thread
172.17.0.1 - - [10/May/2017 14:26:41] "POST http://sqs.eu-west-1.amazonaws.com/123456789012/vlad_facebook_FacebookSearchItemsCount HTTP/1.1" 200 -
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 654, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 713, in finish
self.wfile.close()
File "/usr/lib/python2.7/socket.py", line 283, in close
self.flush()
File "/usr/lib/python2.7/socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
172.17.0.1 - - [10/May/2017 14:26:44] "POST http://sqs.eu-west-1.amazonaws.com/123456789012/vlad_facebook_FacebookWeeklyEmail HTTP/1.1" 200 -
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 596, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 654, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 713, in finish
self.wfile.close()
File "/usr/lib/python2.7/socket.py", line 283, in close
self.flush()
File "/usr/lib/python2.7/socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe

This happens when I stop my app.

[Vladimir Orany]

more or less we're checking the list of queues, create queue if missing and than repeatedly check if there are some messages in the queue. only non-standard thing I can see is setting VisibilityTimeout to 1500 and querying only one last message. I'll try to investigate further.

[Waldemar Hummer]

"This happens when I stop my app." - Does it mean that the exceptions in the output above occur only AFTER you terminate LocalStack? What happens before you terminate it - is there any relevant debug logging?

It is a bit hard to answer the question "how to debug failing features" in general. There is currently no "recommended" way of actually debugging the application. What you can do is either:

1. Provide a minimal test case, including all the relevant client-side code, which allows us to reproduce the problem and find a solution for you.
2. Check out the code from the repository, start LocalStack via make infra, and start debugging yourself.

For option 2, there is actually two more alternatives:

• 2.a. Use your IDE (e.g., IntelliJ or Eclipse) to connect LocalStack to a Python debugger that allows you to set breakpoints etc.
• 2.b. Debug the "old and dirty" way using print(...) statements in the code that help you figure out what is going on and what is going wrong.

I would probably recommend option 2.b because it is the easiest to set up and has the least dependencies, but it very much depends on your personal preferences how you want to take this further. Again, if you provide a reproducible test case, I'd be happy to look into it, but currently you have not provided enough information to reproduce the issue. Thanks

[Vladimir Orany]

No, the main issue happens when we try to poll for messages from the queue. The queues are created with VisibilityTimeout=1800. The messages are polled with

MaxNumberOfMessages = 1 WaitTimeSeconds = 20 VisibilityTimeout = 0

But I'm not able to reproduce it on minimal project so far.

We're assigning lower priority to this so I've also changed the priority of the issue.

For debugging it would help knowing * which code is responsible for handling SQS (it is not clear enough for me as I don't know much Python) * where are the queue data stored (if they are somewhere)

thanks for you cooperation

[Waldemar Hummer]

A good starting point for debugging the API calls is the localstack.mock.proxy package, which contains proxy listeners/interceptors for a subset of the APIs. If you combine it with the DEBUG=1 flag, you should be able to inspect the output from the service processes running in the background. Sometimes it is also beneficial to edit the code of the third-party backend services directly, e.g., for moto services add debug/print statements to .venv/lib/pythonX.Y/site-packages/moto/....

Since I'm not entirely sure how to tackle this request at this point, I'm closing this ticket for now, to have a clean list of backlog tickets that have a clear scope. If you have any concrete idea of how debugging should be facilitated/improved, feel free to re-open with your suggestions. Thanks