}APQ.UX.Server.Maintenance locks up after 2023.05 upgrade

[Wei Wang]

After upgrading to the 2023.05, there appears to be a bug where the “}APQ.UX.Server.Maintenance” TI stalls and never finishes running. Initially noticed this when running the upgrade TI which calls “}APQ.UX.Server.Maintenance”.

It seems to be caused by the section below, which creates a new subset based on Recent Visits. If I take the MDX and try to evaluate it in Arc directly, it creates a thread that never finishes running. On Apliqo_Demo, the issue is not present but on our actual content stores using CAM (after upgrading from 2023.02), it would always get stuck there. For now, we’ve disabled the creation of that subset in the TI.

‌

‌

[Scott Wiltshire]

Hi Wei, Obviously we haven't come accross this in testing or at any other customer to date so it would be good to have some more details or even a full zip of an affected content store to analyse.

Was the upgrade doneonline or offline? Is the upgrade complete? (as this feature relies on some new measures and rule changes in the }APQ UX UserFavorite cube) Is user activity logging enabled in default constants?

[Scott Wiltshire]

Hi Wei, any update? To analyse this I really need to see an example of a CS that has this issue.

[Wei Wang]

hi Scott, thanks for following up and sorry about the delay. We did the offline upgrade and everything completed succesfully.

I did a bit more digging today and found that the root culprit seems to be the C-level ConsolidateMax rule (}APQ UX App Log cube) on the new “VisitTimeStamp” element. The MDX query was ultimately filtering on the “LastVisit” measure element of the “}APQ UX UserFavorite” cube which pulls from the “VisitTimeStamp”. If I disable the ConsolidateMax rule, there is no lock up. In fact, even trying to load a C-level view with this “VisitTimeStamp” in the “}APQ UX App Log” cube causes a lockup. Potentially its just a matter of ConsolidateMax calculating super slow but I waited up to 10 minutes for the query to finish w/ no luck.

Perhaps this is happening because of the number of UX screens we have (i.e., 5000+ elements on the }APQ UX App dim)? Let me know if this new info helps replicate the issue.

‌

‌

[Scott Wiltshire]

Hi Wei,

‌

Wow that’s crazy that you have 5000 elements in your UX App dimension. In the demo application there are currently around 250 (and growing with each release as the new feature screens are added) which I would have already thought is much bigger in terms of screen count compared to what a “normal” customer implementation would be. Are your users making use of private views and that’s where all the screens are coming from? Or are these all public objects?

Can you try changing the flag value for the ConsolidatedMax function to 2 or 3? (This should switch on sparse consolidation for the function and ignore zeros). I already made this change in the rule for the next FP. If that fixes it then great. If not then I might need to look at other ways to optimize; like skipping the calculation if ElLev > 4, or only calculating for view & dashboard.

‌

It would still be really helpful to have a copy of the CS for testing! (Don’t need the reporting DBs as wouldn’t actually connect the CS to UX, just internal testing to see if there are any performance problems with such a large app dimension.)

‌

As a further update on this I tested on a copy of our CS used for unit testing of UX builds. This has hundreds of test cases and consequently hundreds of screens and about 5200 elements in the UX App dimension, which seems very comparable to your database with the locking up issues. Although the My recent pages and My most visited pages subsets are a little slow to evaluate (about 2-3 sec) there’s nothing like a never endling query and the maintenance process completes without issue. I also tested for clearing all data in the UX App Log cube and deleting the subsets in question to make sure there was no issue withthe server being unable to register an empty subset which could have been responsible but the empty dynamic subsets get created just fine without locking. So I see no way to progress without your specific CS as it must be something quite specific to your instance.

[Wei Wang]

Thanks for doing the extensive testing Scott! I modified the ConsolidateMax to use “2” instead of “1” and it seems to work fine now. If you’ve already made the update in the upcoming v2023.05 FP1, then that should resolve this issue. Most of our elements are public objects (I guess there’s just a lot of wizard substeps and toolbar pop-up windows).

On a related matter, in the “}APQ UX App Log” cube, we’ve always had to manually tweak the “Activity” C-level rule after each upgrade. As it currently stands, the ConsolidateChildren seems to be overstating the usage (when we look at monthly or annual figures) since it is on both Client and Time. We would change it to:

ConsolidateChildren( '}APQ UX Client' )

Wanted to bring this up to see if you would consider modifying the rule as such in the core Apliqo package.

‌

‌

[Scott Wiltshire]

Hi Wei,

The ConsolidatedMax flag is changed to 3 in 2023.05 FP1. I will also change the ConsolidateChildren rule to include client.

[Wei Wang]

great, thanks Scott!