Loads of Warnings in Jira

[Nikhil]

Hi Fidel,

Seems like after upgrading plugin to v 2.2.45, we have been getting a lot of warnings in Catalina.out log file.

2017-12-08 01:43:41,207 Caesium-1-2 WARN anonymous Koi [c.a.mail.incoming.mailfetcherservice] Koi[10504]: Unable to create issue with message. com.atlassian.jira.exception.CreateException: Error occurred while creating issue through workflow: at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssue(DefaultIssueManager.java:512) at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssueObject(DefaultIssueManager.java:604) at com.atlassian.jira.issue.managers.RequestCachingIssueManager.createIssueObject(RequestCachingIssueManager.java:209) at com.atlassian.jira.service.util.handler.DefaultMessageHandlerContext.createIssueWithIssueManager(DefaultMessageHandlerContext.java:248) at com.atlassian.jira.service.util.handler.DefaultMessageHandlerContext.createIssue(DefaultMessageHandlerContext.java:166) at com.atlassian.jira.service.services.mail.DelegatingMessageHandlerContext.createIssue(DelegatingMessageHandlerContext.java:43) at com.atlassian.jira.plugins.mail.handlers.CreateIssueHandler.handleMessage(CreateIssueHandler.java:267) at com.atlassian.jira.plugins.mail.handlers.CreateOrCommentHandler.handleMessage(CreateOrCommentHandler.java:133) at com.atlassian.jira.service.services.mail.MailFetcherService$1.process(MailFetcherService.java:376) at com.atlassian.jira.service.services.mail.MailFetcherService$MessageProviderImpl.getAndProcessMail(MailFetcherService.java:255) at com.atlassian.jira.service.services.mail.MailFetcherService.runImpl(MailFetcherService.java:366) at com.atlassian.jira.service.services.file.AbstractMessageHandlingService.run(AbstractMessageHandlingService.java:229) at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:61) at com.atlassian.jira.service.ServiceRunner.runService(ServiceRunner.java:62) at com.atlassian.jira.service.ServiceRunner.runServiceId(ServiceRunner.java:44) at com.atlassian.jira.service.ServiceRunner.runJob(ServiceRunner.java:32) at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153) at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118) at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:438) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:462) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:390) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34) at java.lang.Thread.run(Thread.java:745) Caused by: [InvalidInputException: [Error map: [{}]] [Error list: [[]]] at com.fca.jira.plugins.workflowToolbox.MathExpressionsValidator.validate(MathExpressionsValidator.java:78) at com.atlassian.jira.workflow.SkippableValidator.validate(SkippableValidator.java:45) at com.opensymphony.workflow.AbstractWorkflow.verifyInputs(AbstractWorkflow.java:1466) at com.opensymphony.workflow.AbstractWorkflow.transitionWorkflow(AbstractWorkflow.java:1167) at com.opensymphony.workflow.AbstractWorkflow.initialize(AbstractWorkflow.java:606) at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssue(DefaultIssueManager.java:506) ... 28 more

Can you please help us in resolving this as this is the reason for Jira running very slow and affecting client deliverable.

Nikhil

[Nikhil]

Hello @fcarmario ,

Can you please help us to stop these warnings? Is updating to v2.2.45 causing this issue as this was working well in v2.2.40.

Nikhil

[Fidel Castro Armario]

Hi @nikhil_dsv,

Can you, please, attach the configuration of the "Boolean validator with math, date-time or text-string terms" validation which is causing the error messages? I think it's in "Create Issue" transition of one of your workflows.

Thank you.

[Nikhil]

Attaching the same. The same Validator was working well before we updated.

Can you please come online on Skype so that we can clear things off pretty soon?

Nikhil

[Fidel Castro Armario]

Hi @nikhil_dsv,

Please, try setting parameter "Message to show when validation fails".

[Nikhil]

Setting as "Failed due to Validator mismatch"

[Nikhil]

Now below is the error:

2017-12-08 05:43:13,281 Caesium-1-2 WARN anonymous Koi [c.a.mail.incoming.mailfetcherservice] Koi[10504]: Unable to create issue with message. com.atlassian.jira.exception.CreateException: Error occurred while creating issue through workflow: Failed due to Validator mismatch at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssue(DefaultIssueManager.java:512) at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssueObject(DefaultIssueManager.java:604) at com.atlassian.jira.issue.managers.RequestCachingIssueManager.createIssueObject(RequestCachingIssueManager.java:209) at com.atlassian.jira.service.util.handler.DefaultMessageHandlerContext.createIssueWithIssueManager(DefaultMessageHandlerContext.java:248) at com.atlassian.jira.service.util.handler.DefaultMessageHandlerContext.createIssue(DefaultMessageHandlerContext.java:166) at com.atlassian.jira.service.services.mail.DelegatingMessageHandlerContext.createIssue(DelegatingMessageHandlerContext.java:43) at com.atlassian.jira.plugins.mail.handlers.CreateIssueHandler.handleMessage(CreateIssueHandler.java:267) at com.atlassian.jira.plugins.mail.handlers.CreateOrCommentHandler.handleMessage(CreateOrCommentHandler.java:133) at com.atlassian.jira.service.services.mail.MailFetcherService$1.process(MailFetcherService.java:376)

The lines entered are displayed in the 2nd line of the error above.

Nikhil

[Nikhil]

Logging "c.a.mail.incoming.mailfetcherservice" to WARN/ERROR did not work. The logs still flooding.

[Fidel Castro Armario]

Which version of JIRA are you using?

[Nikhil]

Hi @fcarmario,

It's version 7.3.0

[Nikhil]

If I remove this Validator, the warnings stops and issues gets generated which we do not want to.

Nikhil

[Fidel Castro Armario]

When you remove the validator, does JIRA stop running slow?

How many issues per minute, approximately, are you blocking from being created due to that validator?

[Nikhil]

Jira is working fine now, probably it's been 5 to 6 hours and it was very slow before that. I am not sure about the count, but when I removed this Validator for a minute or so, 152 issue got created in Jira. So, enabled back the validator. Can this be the reason for very high Connection Pool in Jira, due to which Jira was very slow? it was like this and I am afraid if this repeats.

Nikhil

[Nikhil]

Hi Fidel,

Did you find any resolution?

Nikhil

[Fidel Castro Armario]

I'm trying to reproduce your problem. I don't achieve to get the same error message you are getting in your log file.

Can you, please, tell me the logging level? It's at Administration > System > Logging and profiling.

[Nikhil]

Logging level of "c.a.mail.incoming.mailfetcherservice"?

c.a.mail.incoming.mailfetcherservice ERROR

[Fidel Castro Armario]

Please, tell me the default logging level.

[Nikhil]

The Default has WARN

[Fidel Castro Armario]

Hi @nikhil_dsv,

I have managed to obtain the same message in my log file. It happens also with JWT 2.2.40, so that's not a problem due to the update to JWT 2.2.45.

It's indeed normal behavior: when an issue can't be created through email due to a validation unsatisfied, JIRA throws that message to the server's log file.

You can avoid the error message by setting the default logging level to ERROR.

I don't think your performance problems are caused by JWT, except by the fact that you can eventually have bursts of emails needing to be processed by JIRA, and maybe that can be affecting the performance at some point. Can you, somehow, reduce the number of emails generated by Naggios or your monitoring software?

[Nikhil]

Hi @fcarmario ,

So, setting the Default to "Error" doesn't affect other incoming mails right? What change does this bring? Probably yes, the app was very slow for about 3 to 4 hours and became normal.

Nikhil

[Nikhil]

We have killed the Nagios process completely, but the logging is still happening.

[Nikhil]

Making Default Logging level to Error did not work either. It was still logging same warnings.

Nikhil

[Fidel Castro Armario]

Hi @nikhil_dsv,

You are right. Changing the logging leven doesn't eliminate the messages in the log file.

I have also found out that the messages are not being eliminated from the mailbox, and JIRA keeps trying to create the same issues periodically and throwing messages to the log file continually. This fact also explains the performance issues, since after some time you have a huge number of emails accumulated in the mailbox which are being periodically processed by JIRA.

I don't consider it a bug of JWT, but a problematic behavior of JIRA itself. Now, I see that it's not a good idea to block issue creation by emails through validations. We have 2 alternatives:

1) Use filter in the mailbox to automatically remove emails which satisfy the following condition:

From contains "nagios" AND subject doesn't contain "DOWN"

2) Add "Copy a parsed text to a field" post-function to your "Create Issue" transition with the following configuration:

• Target field: Issue Status
• Parsing mode: basic
• Text to be parsed: Cancelled
• Conditional execution:

%{00006} ~ "nagios" AND %{00000} !~ "DOWN"

I'm assuming that you have a "Cancelled" status in your workflow, and that you have a transition from "Open" status (actually from first status of your workflow) to "Cancelled" status.

With the second solution, what we do is to automatically cancel the issues just after their creation.

Don't forget to remove the validation and to clean all the old emails accumulated in your mailbox.

[Nikhil]

In the first solution, we cannot filter the issues as it is Roundcube mail box but will try. Second solution seems good but one kind of issues we blocked were alerts from Nagios if Down and other were from Nagios VPN tunnel that created so much of trouble today. You are right, blocking them created issues so we have to either allow them to get them created and cancel oŕ delete them?

[Fidel Castro Armario]

We cannot delete issues using a post-function, since JWT doesn't provide a post-function for deleting issues (I find it too dangerous). You can cancel them (moving to "Cancelled" status), and periodically delete all the cancelled issues.

[Fidel Castro Armario]

Hi @nikhil_dsv,

Can we close the issue?

[Nikhil]

Yes, thank you again.

Nikhil

[Nikhil]

Will reopen when we have any doubts regarding this.