Sassy is depressed and suicidal

[Joshua Petry]

/meme genome soldier |!|

The above results in a lot of odd results, with a red background, such as:

• I think you ought to know I'm feeling very depressed.
• I'm just trying to die.
• I've seen it. It's rubbish.
• Sorry guys, I can’t do that right now. All my circuits are currently engaged on solving a different problem. Now I know this is very unusual, but it is a very difficult and challenging problem, and I know that the results will be one we can all Share and Enjoy.

If it's related, /img genome soldier results in a dead image.

[Schoenhofen, Kelly]

This is apparently some sort of "Sassy is down" text. It's happening right now for us; how can we tell when new versions of Sassy go in, or its status? Or how do we report it's down? https://status.hipchat.com/ doesn't show bot status.

With Sassy down, we're down.

[Robert Bergman]

This text usually occurs when Sassy throws an unexpected error processing a given command, or one of the external services its communicating with is erroring out. @kschoenhofen can you give some details on what commands you're running that are returning that text? Sassy is running normally right now, and I don't see any relevant errors in the last 24 hours worth of logs.

[Schoenhofen, Kelly]

We're seeing this today:

Sassy is enabled in our rooms, and commands like /weather work just fine, but we can't get any /meme to work.

[Robert Bergman]

Thanks, I am seeing this now too for /meme. I'll look into it as soon as I can.

[Schoenhofen, Kelly]

Super. We have complete work stoppage in our engineering staff right now as a result.

[Robert Bergman]

This seems to have been caused by a temporary problem with the memecaptain API that has since resolved itself.

[Schoenhofen, Kelly]

Thanks. It looks like it was rectified by 5AM CDT April 29th when an engineer noticed it was working again; the very next line was: http://v1.memecaptain.com/67651b.jpg.

[Joshua Petry]

She's going crazy again.

[Schoenhofen, Kelly]

Yep. We noticed a few hours ago. Sigh.

[Robert Bergman]

Sorry. This is due to the meme service Sassy uses having availability issues. I've updated Sassy's error messages to be clear about what's wrong when this happens. I may evaluate some other meme generator APIs, given how flaky memecaptain has been lately.

[Schoenhofen, Kelly]

Hey. I found memecaptain's github repo and opened up an issue ~4 hours ago. They just responded, apparently their old v1 server/vm/instance/docker (whatever) had fallen over and nobody told them.

Sigh.

https://github.com/mmb/meme_captain_web/issues/3

Hopefully they will reply with info on the v2 api we should be using (at the time I am writing this, they haven't responded after I asked my question about APIs)?

Or have you done some research and you found something you want to move to already?

[Robert Bergman]

No, I haven't had a time to look at it yet. I didn't realize they had a new API version we should move to either. Let me know if you hear back on that. If you want to PR an upgrade, I'd be happy to verify and deploy if it works.

I have a suspicion that the service might crash if large gifs are fed to it, but don't want to test it lest it goes offline again ;) A better solution might be for me to try to standup a worker service just for Sassy running their generator so that I can bounce it myself when needed. If it continues to go down I might give that a try.

[Robert Bergman]

Closing this since memecaptain is responding again. Please open a new issue for any recurring /meme issues.

[Schoenhofen, Kelly]

Adding a comment - the memecaptain people are finishing up their new API and are removing the v1 API shortly, FYI.

https://github.com/mmb/meme_captain_web/

[David Escalona]

Woops, yeah, looks like something is wrong. Hipchat reports that memecaptain API is down (maybe it is already terminated) and a week without /meme is a sad week :(

[Robert Bergman]

A fix for the /meme command was deployed yesterday.

[Schoenhofen, Kelly]

Yes, it's worked out well. I updated issue #11 when it was deployed, not this one.

The only problem is the performance at times isn't very robust, and I've seen a timeout already. Most of the time it's fairly responsive.

Not complaining ;) just noting. I don't know where the slowdown is; I assume it's on the memecaptain side.

[Robert Bergman]

The way the new v2 api works is that we have to submit a request to begin generating the meme and then poll for the result. The way I've set up the polling for now is to check for a result 10 times with a linear back off over a total of about 15s ± request overhead. When I tested yesterday, I could only get the timeout notice when trying to memeify (enmemeinate?) some animated gifs, FWIW. As I understand it (via @mm6), the polling requests should be low overhead on the meme captain side, so we ought to be able to adjust the polling rate and duration to get fewer timeouts if the current polling model proves to lead to frequent timeouts. Any information you can provide about the frequency of timeouts and what images they're occurring for would be helpful.

[mm6]

Some of these may be errors that appear to be timeouts. Currently errors (like failing to fetch a source image) are retried until they time out. I will look into a way to give the client better error information.

@rbergman There may also be an issue with the client not polling when it creates source images. It should poll then too. This has exposed a bug in the API where it will try to create a meme even when the source image is not finished yet. I will fix that.

I'll be happy to investigate any images that failed or took a long time to return from the server side.

[Robert Bergman]

Ah ok... I saw you had the polling comment up near the source image creation too -- I should have asked, but it looked unnecessary and it didn't cause any issues without it in my testing, so I dropped it. It should be relatively simple to add that back in, though.