Very long loading times for Gadget with around 6k tickets

[Philipp Sendek]

Hi there,

I’m a colleague from Kevin who recently opened https://bitbucket.org/StonikByte/great-gadgets-add-on/issues/84/performance-issue-in-jira-data-center which was already resolved.

We have upgrade to version 1.21.2 of Great Gadgets already, but still experience very long loading times on certain gadgets - even though they have been loaded already shortly before and therefor should be cached.
Here is a screenshot of the configuration of one of those gadgets (One particular dashboard has 10 of them with different filters and configurations.

‌

The selected filter has the following JQL:
issuetype = Story AND labels = "#dpw-issue" AND status not in (Cancelled, Canceled, "Finally Rejected", Rejected) AND fixVersion = DPW_REF-1.0

It returns 6770 issues. The instance itself has 200k issues.

My specific questions would be:

• Do you have any known issues with such an amount of issues at once and is the long loading time also directly related to the Series that are displayed?
• And if so, do you see any potential to improve the performance in these areas or tune the Caching, so that it improves loading time also in situations like this, where it doesn’t seem to work for some reason?
• Do you have an estimate on whether the loading time also means that the gadget is consuming high CPU on the server side?

As we can’t ask our client to reduce the amount of tickets they want to view, we need to find a way to improve the situation.

Thank you for your insights.

Regards,
Philipp

‌

[Danut M [StonikByte]]

Hi Philipp,

‌

Thanks for posting.

‌

Let’s take this particular gadget, from your example. It a Release Burnup Burndown Chart Gadget.

How long does it take to load?

During the gadget loading, what message do you see:

• “Cached data found. Processing latest changes...“, or
• “Processing... It might take a bit longer this time because there is no cached data.“

Are there any errors in the Jira log file? Like these, for example: Error updating the cached results for key or Error updating the cached results for key?

‌

Thank you,

Danut Manda

[Philipp Sendek]

Hi Danut,

thank you for getting back to me that quickly.
Taking the whole dashboard, I can say it took roughly 1 minute to load all gadgets except for this particular one.
This particular one went through the different phases with the following durations:

• Scheduling for Processing 13 seconds
• Scheduled for processing 4 sec
• Looking for cached data 6 seconds
• Cached data found, processing latest changes 4 Min

While opening the dashboard, there are no log entries for great gadget in the atlassian-jira.log.

The system is running on a VM with 8 vCPUs and currently 2-4 vCPUs are occoupied due to heavy user load (users booking time as it’s the end of the month), so we know that the current load of the system is significant.
However, a dashboard gadget to take 4 minutes to load seems very unreasonable for me.

Regards,
Philipp

‌

[Danut M [StonikByte]]

Hi Philipp,

‌

Thanks. The caching seems used.

I see that the gadget is configured to use intervals of 4 weeks. The gadget does not cache the current interval (which is incomplete), so it recalculates for the issues updated since the last complete interval. In your case it might be a large number of issues.

Please try setting Intervals of 1 week or, even better, 1 day. Let me know if this makes any significant difference.

Danut

[Philipp Sendek]

Hi Danut,

thank you, I will recommend it to the team.
In case they certainly need this timeframe, are there any other things we can improve?
Or put differently: Would you be able to share the issue data that the app runs through? Maybe we have some issues with a large history or something, that extends the whole process - for instance, due to a bug in an app, we had 2 issues in the past with 50k (!) change history entries, that alone took a few minutes to load when opening them.

I’d be curious how we could troubleshoot the issues to find such blockers. Or do you think it’s just the combination of 6k issues with 4 weeks timeframes?

Regards,
Philipp

[Philipp Sendek]

Hi Danut,

Would you be able to have a look into my questions in the last comment?
I would highly appreciate it!

Regards,
Philipp

[Danut M [StonikByte]]

Hi Philipp,

The app uses the change log events of the issues from the filter to determine the changes (status, estimate, etc) along the time. Large issue history events might affect the gadget loading.

Did you try the workaround proposed? If it does not resolve we should continue the troubleshooting. We could see where it takes much time to process, by enabling debug logging.

Danut.

‌

‌

[Philipp Sendek]

Hi Danut,

thanks for this information, that helps us understanding and shed some light on some things.

So regarding the workaround: I have create a copy of the dashboard a while ago and have just set all gadgets to interval of 1 week.
The result is still, that after the page loaded and the gadget “frames” appear, it takes up to 3,5 minutes for the particular gadget (config above) to show the result. This time it again didn’t pull data from cache but processed the data from scratch, which of course takes long than caching.

When I simply refresh the page, the mentioned gadget is one of the first that’s loaded with roughly 20secs. The others follow after 5-10 additional seconds. So all gadgets on this dashboard are loaded after about one minute.

Getting back to your information regarding the history:
We have one particular issue in this JQL that has 18k hours logged against it with a history of 1660 entries.
Other issues with more then 100 history entries in the JQL are:
issue: L2O-5102: 206
issue: L2O-4270: 205
issue: L2O-4213: 204
issue: L2O-4207: 249
issue: L2O-4190: 294
issue: L2O-4077: 233
issue: L2O-4069: 232
issue: L2O-3442: 277
issue: L2O-3293: 217
issue: L2O-2725: 219
issue: DPWP2P-300: 252
issue: DPWP2P-299: 276
issue: DPWMDM-2: 1660 (high value as this is an issue for time tracking with 18k hours logged to it)
issue: DPWEDW-2: 289
issue: DCCM-255: 602

Is it fair to assume, that this is causing the delay? How many entries do the issues in your test data have?

Maybe you can run some tests with similarly sized issues to see how your gadgets perform there.

Regards,
Philipp

[Philipp Sendek]

Hi Danut,

I reran out check to see how many more issues there are when I set the threshold to 100 entries in the change history.
It turns out, it’s 183 issues in total (out of a little more than 6000):
issue: S2R-4973: 102
issue: S2R-4952: 109
issue: S2R-4860: 108
issue: S2R-4749: 105
issue: S2R-4729: 128
issue: S2R-4688: 156
issue: S2R-4354: 102
issue: S2R-4303: 116
issue: S2R-4243: 116
issue: S2R-4163: 128
issue: S2R-4141: 107
issue: S2R-3821: 107
issue: S2R-3404: 101
issue: S2R-3403: 140
issue: S2R-3402: 125
issue: S2R-3367: 103
issue: S2R-3361: 118
issue: S2R-3354: 104
issue: S2R-3335: 111
issue: S2R-3298: 129
issue: S2R-3262: 102
issue: S2R-3184: 106
issue: S2R-3182: 146
issue: S2R-3180: 103
issue: S2R-3179: 149
issue: S2R-3153: 113
issue: S2R-2997: 114
issue: S2R-2985: 106
issue: S2R-2954: 102
issue: S2R-2627: 102
issue: S2R-2608: 104
issue: S2R-2414: 114
issue: S2R-2411: 110
issue: S2R-2406: 129
issue: L2O-5288: 122
issue: L2O-5127: 103
issue: L2O-5124: 125
issue: L2O-5102: 206
issue: L2O-5098: 142
issue: L2O-5091: 102
issue: L2O-5045: 110
issue: L2O-5044: 106
issue: L2O-4968: 112
issue: L2O-4939: 111
issue: L2O-4910: 188
issue: L2O-4811: 136
issue: L2O-4774: 164
issue: L2O-4772: 106
issue: L2O-4757: 114
issue: L2O-4732: 105
issue: L2O-4713: 107
issue: L2O-4673: 112
issue: L2O-4638: 129
issue: L2O-4553: 117
issue: L2O-4549: 191
issue: L2O-4520: 122
issue: L2O-4512: 103
issue: L2O-4442: 125
issue: L2O-4389: 106
issue: L2O-4346: 110
issue: L2O-4322: 117
issue: L2O-4270: 205
issue: L2O-4263: 135
issue: L2O-4238: 165
issue: L2O-4237: 113
issue: L2O-4220: 109
issue: L2O-4219: 163
issue: L2O-4218: 118
issue: L2O-4217: 121
issue: L2O-4213: 204
issue: L2O-4212: 113
issue: L2O-4207: 249
issue: L2O-4190: 294
issue: L2O-4183: 119
issue: L2O-4152: 112
issue: L2O-4124: 108
issue: L2O-4080: 124
issue: L2O-4077: 233
issue: L2O-4070: 108
issue: L2O-4069: 232
issue: L2O-4066: 114
issue: L2O-4065: 108
issue: L2O-4063: 181
issue: L2O-4054: 117
issue: L2O-4051: 107
issue: L2O-3994: 118
issue: L2O-3993: 119
issue: L2O-3992: 104
issue: L2O-3987: 178
issue: L2O-3981: 168
issue: L2O-3977: 132
issue: L2O-3973: 146
issue: L2O-3930: 112
issue: L2O-3929: 105
issue: L2O-3920: 122
issue: L2O-3918: 142
issue: L2O-3902: 121
issue: L2O-3900: 145
issue: L2O-3894: 103
issue: L2O-3888: 167
issue: L2O-3884: 112
issue: L2O-3882: 119
issue: L2O-3870: 123
issue: L2O-3869: 109
issue: L2O-3856: 109
issue: L2O-3815: 142
issue: L2O-3768: 118
issue: L2O-3679: 167
issue: L2O-3675: 122
issue: L2O-3660: 139
issue: L2O-3642: 140
issue: L2O-3521: 161
issue: L2O-3512: 115
issue: L2O-3442: 277
issue: L2O-3372: 138
issue: L2O-3362: 103
issue: L2O-3335: 139
issue: L2O-3327: 101
issue: L2O-3293: 217
issue: L2O-3231: 109
issue: L2O-3212: 133
issue: L2O-3172: 105
issue: L2O-3169: 190
issue: L2O-3101: 117
issue: L2O-3065: 104
issue: L2O-3007: 164
issue: L2O-2961: 101
issue: L2O-2952: 137
issue: L2O-2936: 146
issue: L2O-2840: 131
issue: L2O-2746: 146
issue: L2O-2745: 154
issue: L2O-2744: 115
issue: L2O-2729: 112
issue: L2O-2725: 219
issue: DPWR2R-3085: 146
issue: DPWR2R-2793: 128
issue: DPWR2R-212: 104
issue: DPWR2R-209: 103
issue: DPWR2R-193: 103
issue: DPWR2R-146: 110
issue: DPWP2P-951: 118
issue: DPWP2P-676: 151
issue: DPWP2P-501: 113
issue: DPWP2P-353: 107
issue: DPWP2P-300: 252
issue: DPWP2P-299: 276
issue: DPWP2P-295: 107
issue: DPWP2P-267: 117
issue: DPWP2P-266: 105
issue: DPWP2P-247: 114
issue: DPWP2P-246: 187
issue: DPWP2P-139: 132
issue: DPWP2P-102: 112
issue: DPWP2P-101: 107
issue: DPWP2P-100: 105
issue: DPWP2P-98: 183
issue: DPWP2P-90: 153
issue: DPWP2P-89: 171
issue: DPWP2P-88: 176
issue: DPWP2P-87: 155
issue: DPWP2P-86: 166
issue: DPWO2C-2712: 126
issue: DPWO2C-2137: 104
issue: DPWO2C-2086: 115
issue: DPWO2C-1884: 106
issue: DPWO2C-657: 190
issue: DPWO2C-4: 113
issue: DPWMDM-3829: 102
issue: DPWMDM-3795: 106
issue: DPWMDM-3463: 129
issue: DPWMDM-796: 126
issue: DPWMDM-4: 105
issue: DPWMDM-2: 1660
issue: DPWF2S-568: 114
issue: DPWF2S-241: 105
issue: DPWF2S-7: 123
issue: DPWEDW-1144: 123
issue: DPWEDW-1143: 104
issue: DPWEDW-1142: 129
issue: DPWEDW-2: 289
issue: DCCM-255: 602

Needless to say, that we cannot purge the histories of those issues, so either the app needs to be able to run through them faster, we might find a way to improve the performance (maybe a database index if there isn’t one already?) or we team needs to live with the current loading time.

Regards,
Philipp

[Danut M [StonikByte]]

Hi Pihlipp,

‌

Thanks for providing all these details.

The issues that have more than 100 change logs events can affect in some extent, especially at the initial gadget loading when the cache is created. But once the cache is created, the can affect only if they were modified/updated in the current interval of the gadget because otherwise they are not processed again.

Can you please exclude the issue with 1600 change log events from the filter and retest to see if it makes any difference?

Also, could you please retest by grouping the data by 1 day (instead of 1 week) in gadget settings? I expect this to reduce the loading time.

‌

Danut

[Philipp Sendek]

Hi Danut,

I have excluded the particular issue from the JQL and the gadget still loads 4,5 minutes, with nearly all the time in “Processing... It might take a bit longer this time because there is no cached data.“.

I have set the interval to 1 day now and switching back and forth between the original dashboard (set to 4 weeks, I think) and my copy (set to 1 day) makes me believe my dashboard is loading faster.
However, right now, both dashboards are completely loaded within a few seconds. I suppose because there’s no change in data right now and it can simply load from the cache.

To understand what I just did: What is the technical difference between 1 week and 1 day? Is clustering the data by week taking more time?

Regards,
Philipp

[Danut M [StonikByte]]

Hi Phillip,

The cached is saved only for the completed time intervals, the current interval not being cached.

For example, you have the gadget configured to group data by 1 week and the release start date is 01-NOV-2020. The time intervals in the chart are [01-07] , [08-14], etc . If you display the gadget Today (12-NOV), this date is in the second time interval [08-14], which is the current (incomplete) interval. The cache contains data only for the completed intervals ([01-07] in this example) and when you load the gadget it will process all the issues from the filter created/modified since 08-NOV.

So when the gadget is configured to group data by 1 week, besides the caching data, it processes issues from current interval, which can be up-to ~7 days long in the worst case. And if there were many issues updated, it will take longer to load.

When the gadget is configured to group data by 1-day, besides the cached data, it processes only the issues created/updated in the current day. So less issues to process and more cached data, better loading time.

Does it make sense?

My advice is to configure the gadget to group data by 1 day.

Thank you,

Danut Manda

[Philipp Sendek]

Hi Danut,

yes, it makes perfect sense now and I totally understand why we should leave this at 1 Day.

I will forward this recommendation to the customer.
Sorry, if I didn’t concur with setting this earlier. I thought this interval would mean the timeframe of the graph, i.e. it would only display the history of one day.

Regards,
Philipp