SOQL query on ServiceReport object crashes the IDE

[René Görgens]

Hi Scott,

Running the following query hangs the IDE, the UI is frozen and I have to kill the IDE.

SELECT FIELDS(ALL)
FROM ServiceReport
LIMIT 200

The query works without issues in Dev Console.

Log:

2024-03-17 19:41:20,011 [ 42741] INFO - #c.i.w.i.i.j.s.JpsGlobalModelSynchronizerImpl - Saving global entities to files
2024-03-17 19:41:21,799 [ 44529] SEVERE - #c.i.i.i.DataValidators - PSI element is provided on EDT by com.illuminatedcloud.intellij.util.IcUiUtil$1.getData("psi.File"). Please move that to a BGT data provider using PlatformCoreDataKeys.BGT_DATA_PROVIDER [Plugin: com.illuminatedcloud2.intellij]
com.intellij.diagnostic.PluginException: PSI element is provided on EDT by com.illuminatedcloud.intellij.util.IcUiUtil$1.getData("psi.File"). Please move that to a BGT data provider using PlatformCoreDataKeys.BGT_DATA_PROVIDER [Plugin: com.illuminatedcloud2.intellij]
at com.intellij.diagnostic.PluginProblemReporterImpl.createPluginExceptionByClass(PluginProblemReporterImpl.java:23)
at com.intellij.diagnostic.PluginException.createByClass(PluginException.java:89)
at com.intellij.ide.impl.DataValidators.reportPsiElementOnEdt(DataValidators.java:96)
at com.intellij.ide.impl.DataValidators.isDataValid(DataValidators.java:71)
at com.intellij.ide.impl.DataValidators.validOrNull(DataValidators.java:64)
at com.intellij.ide.impl.DataManagerImpl.getDataFromProviderInner(DataManagerImpl.java:245)
at com.intellij.ide.impl.DataManagerImpl.getDataFromProviderAndRulesInner(DataManagerImpl.java:82)
at com.intellij.ide.impl.DataManagerImpl.getDataFromProviderAndRules(DataManagerImpl.java:61)
at com.intellij.openapi.actionSystem.impl.PreCachedDataContext.cacheProviderData(PreCachedDataContext.java:339)
at com.intellij.openapi.actionSystem.impl.PreCachedDataContext.cacheComponentsData(PreCachedDataContext.java:315)
at com.intellij.openapi.actionSystem.impl.PreCachedDataContext.<init>(PreCachedDataContext.java:104)
at com.intellij.openapi.actionSystem.impl.Utils.createAsyncDataContextImpl(Utils.kt:163)
at com.intellij.openapi.actionSystem.impl.Utils.createAsyncDataContext(Utils.kt:153)
at com.intellij.ide.navbar.ide.NavBarIdeUtil.dataContextInner(util.kt:115)
at com.intellij.ide.navbar.ide.NavBarIdeUtil.access$dataContextInner(util.kt:1)
at com.intellij.ide.navbar.ide.NavBarIdeUtil$dataContext$2$1.run(util.kt:99)
at com.intellij.openapi.wm.impl.FocusManagerImpl.lambda$doWhenFocusSettlesDown$3(FocusManagerImpl.java:169)
at com.intellij.util.ui.EdtInvocationManager.invokeLaterIfNeeded(EdtInvocationManager.java:33)
at com.intellij.ide.IdeEventQueue.ifFocusEventsInTheQueue(IdeEventQueue.kt:220)
at com.intellij.ide.IdeEventQueue.executeWhenAllFocusEventsLeftTheQueue(IdeEventQueue.kt:186)
at com.intellij.openapi.wm.impl.FocusManagerImpl.doWhenFocusSettlesDown(FocusManagerImpl.java:165)
at com.intellij.openapi.wm.impl.FocusManagerImpl.lambda$doWhenFocusSettlesDown$2(FocusManagerImpl.java:174)
at com.intellij.openapi.application.TransactionGuardImpl.runWithWritingAllowed(TransactionGuardImpl.java:209)
at com.intellij.openapi.application.TransactionGuardImpl.access$100(TransactionGuardImpl.java:22)
at com.intellij.openapi.application.TransactionGuardImpl$1.run(TransactionGuardImpl.java:191)
at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:843)
at com.intellij.openapi.application.impl.ApplicationImpl$4.run(ApplicationImpl.java:463)
at com.intellij.openapi.application.impl.RwLockHolder.runWithEnabledImplicitRead(RwLockHolder.kt:75)
at com.intellij.openapi.application.impl.RwLockHolder.runWithImplicitRead(RwLockHolder.kt:67)
at com.intellij.openapi.application.impl.ApplicationImpl.runWithImplicitRead(ApplicationImpl.java:1436)
at com.intellij.openapi.application.impl.FlushQueue.doRun(FlushQueue.java:82)
at com.intellij.openapi.application.impl.FlushQueue.runNextEvent(FlushQueue.java:124)
at com.intellij.openapi.application.impl.FlushQueue.flushNow(FlushQueue.java:44)
at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:318)
at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:792)
at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:739)
at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:733)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:761)
at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.kt:695)
at com.intellij.ide.IdeEventQueue._dispatchEvent$lambda$12(IdeEventQueue.kt:589)
at com.intellij.openapi.application.impl.RwLockHolder.runWithoutImplicitRead(RwLockHolder.kt:44)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.kt:589)
at com.intellij.ide.IdeEventQueue.access$_dispatchEvent(IdeEventQueue.kt:72)
at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:355)
at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:354)
at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:793)
at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:354)
at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:349)
at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$1(IdeEventQueue.kt:1014)
at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:106)
at com.intellij.ide.IdeEventQueueKt.performActivity(IdeEventQueue.kt:1014)
at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$7(IdeEventQueue.kt:349)
at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:848)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.kt:391)
at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207)
at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128)
at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117)
at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113)
at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105)
at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)
2024-03-17 19:41:21,800 [ 44530] SEVERE - #c.i.i.i.DataValidators - IntelliJ IDEA 2023.3.5 Build #IU-233.14808.21
2024-03-17 19:41:21,800 [ 44530] SEVERE - #c.i.i.i.DataValidators - JDK: 17.0.10; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o.
2024-03-17 19:41:21,800 [ 44530] SEVERE - #c.i.i.i.DataValidators - OS: Windows 11
2024-03-17 19:41:21,800 [ 44530] SEVERE - #c.i.i.i.DataValidators - Plugin to blame: Illuminated Cloud 2 version: 2.3.0.2
2024-03-17 19:41:21,855 [ 44585] INFO - org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean - Creating Service {urn:partner.soap.sforce.com}PartnerService from class com.sforce.soap.partner.PartnerApi
2024-03-17 19:41:22,164 [ 44894] INFO - #com.illuminatedcloud.intellij.client.ForceComApiClient - JAXB initialization for com.sforce.soap.partner.SessionHeader took 8 ms.
2024-03-17 19:41:22,172 [ 44902] INFO - #com.illuminatedcloud.intellij.client.ForceComApiClient - JAXB initialization for com.sforce.soap.partner.CallOptions took 7 ms.
2024-03-17 19:41:22,176 [ 44906] INFO - org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean - Creating Service {http://soap.sforce.com/2006/04/metadata}MetadataService from class com.sforce.soap.metadata.MetadataApi
2024-03-17 19:41:22,318 [ 45048] INFO - #com.illuminatedcloud.intellij.client.ForceComApiClient - JAXB initialization for com.sforce.soap.metadata.SessionHeader took 28 ms.
2024-03-17 19:41:22,342 [ 45072] INFO - #com.illuminatedcloud.intellij.client.ForceComApiClient - JAXB initialization for com.sforce.soap.metadata.CallOptions took 24 ms.
2024-03-17 19:41:22,343 [ 45073] INFO - org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean - Creating Service {http://soap.sforce.com/2006/08/apex}ApexService from class com.sforce.soap.apex.ApexApi
2024-03-17 19:41:22,364 [ 45094] INFO - #com.illuminatedcloud.intellij.client.ForceComApiClient - JAXB initialization for com.sforce.soap.apex.SessionHeader took 3 ms.
2024-03-17 19:41:22,366 [ 45096] INFO - #com.illuminatedcloud.intellij.client.ForceComApiClient - JAXB initialization for com.sforce.soap.apex.CallOptions took 2 ms.
2024-03-17 19:41:22,368 [ 45098] INFO - org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean - Creating Service {urn:tooling.soap.sforce.com}ToolingService from class com.sforce.soap.tooling.ToolingApi
2024-03-17 19:41:22,477 [ 45207] INFO - #com.illuminatedcloud.intellij.client.ForceComApiClient - JAXB initialization for com.sforce.soap.tooling.SessionHeader took 22 ms.
2024-03-17 19:41:22,500 [ 45230] INFO - #com.illuminatedcloud.intellij.client.ForceComApiClient - JAXB initialization for com.sforce.soap.tooling.CallOptions took 23 ms.
2024-03-17 19:41:25,019 [ 47749] INFO - #c.i.w.i.i.j.s.JpsGlobalModelSynchronizerImpl - Saving global entities to files

[René Görgens]

FIELDS(ALL) contains the base64-encoded DocumentBody: https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_servicereport.htm

• The dev console presents only a reference to the body’s API endpoint in DocumentBody and leaves DocumentLength blank
• IC fetches the actual body and retrieves DocumentLength

The binary size is in the range 30 to 600 KB, but there were 3 large outliers of > 30 MB each, which I have deleted. Now the query works (against 25 total rows).

Could any kind of warning be applied when querying a dataset containing binary data?

[Scott Wells]

The issue here is that the API call to execute the SOQL query is long-running in Salesforce. It would time out eventually (if memory serves, the default timeout is 5 minutes), but in the interim the calling thread blocks. I do agree that the calling thread should not be the EDT, though. I'll take a look at why that's the case and whether it can easily be offloaded onto a proper worker thread.

[Scott Wells]

Also, the SOQL Query tool window does prompt the user if a query includes Blob fields without reasonable constraints, e.g.:

‌

That’s also configurable in SOQL Query settings:

‌

I don’t have ServiceReport in my test orgs. What is the data type of that field? If it’s Blob, I would expect IC2 to have prompted before executing the query.

[René Görgens]

ServiceReport is a Field Service object. In order to enable Field Service, (in a Developer Edition) navigate to Setup > Field Service Settings and enable Field Service. Then you have access to ServiceReport.

My query has a LIMIT clause but the EDT thread blocking (if that’s the reason for the screen freeze) occurred nonetheless due to large Blob values on a few records. So maybe even constrained queries for Blob fields should be validated?

[Scott Wells]

After digging in a bit, the time is actually not being spent on the query itself but rather on a multi-byte character set-aware cell renderer processing Base64-encoded Blob data unnecessarily. I’ve committed a fix for that for the next build.

[Scott Wells]

Delivered in 2.3.0.4.

[René Görgens]

Retested with 2.3.0.4

• Added 3 ServiceReport records with a Blob > 30 MB each, 28 records overall
• This query takes 10+ seconds:

SELECT Id, ServiceReportNumber, DocumentLength, Status
FROM ServiceReport

• This query (i.e. including the Blob field) takes 60+ seconds:

SELECT FIELDS(ALL)
FROM ServiceReport
LIMIT 200

• Data window behaviour

  • Scrolling to the right becomes jumpy/sluggish once the Blob field is pulled into view (up to 10 seconds for a right scroll increment)
  • While the Blob field is in the data viewport, the whole IDE is extremely sluggish (5 seconds between clicks/reactions e.g. to set the cursor)

  • Once the Blob field is moved completely out of the data viewport or the SOQL window hidden, normal IDE behaviour is restored

    • Showing the SOQL window again takes 10 seconds

=> I think there’s still a screen rendering issue; although only a few characters of the Blob are actually visible in the data window, it impedes the UI in various ways.

The original issue is solved, the UI remains reactive during query execution.

[Scott Wells]

Yeah, unfortunately there’s only so much I can do. I’ve moved to the most efficient table cell renderer available that can still show MBCS data (which, admittedly, is not found in Base64-encoded Blob text). If you query a large amount of Blob data – in your example, ~100MB which are going to be ~133MB when Base64-encoded – it’s going to be sluggish. At this point the UI doesn’t lock up any longer because the amount of work being done on the critical EDT is minimized, but there’s no getting around it being sluggish. You can, of course, either omit those fields from the query (it’s questionable why they’re needed for ad hoc queries) or export to CSV and view the data in an external tool that’s more tuned for large data sets.

FWIW, I did consider truncating large Blob fields (and text in general since large text fields can be up to 131K in length), but that can lead to other issues if users query said data and then try to copy it to the clipboard for use elsewhere.

[René Görgens]

Thanks for the explanation!

Yeah, it’s kind of unclear why the Blob would need to be in the query results in the first place. I came across it accidentally let’s say, by using FIELDS(ALL) for diagnostic purposes – or because I was lazy.