Protection against huge SOQL query results

[Marcel Pękacki]

It happened to me many times that when using SOQL Query functionality I selected all records in the org without realizing how many records there actually are. When there is a really large amount of records (thousands), the IDE essentially freezes when trying to parse such a huge response. In such case I either have to kill the process or wait a really long time until the response actually parses.

I'm not sure if it event would be possible, but it would be great if Illuminated Cloud 2 could detect if the response coming from SOQL query is really huge and then display a warning with a confirmation message. That would make me a lot less conscious when using SOQL query functionality

Alternatively, huge responses could be paginated or rendered lazily to improve performance.

[Scott Wells]

This is closely related to #876 which is on my relatively near-term backlog. There are several ways that a SOQL query can go south, whether due to a large number of rows, a complex nested relational structure, and/or queried fields that contain extremely large data. I'll definitely add something to the SOQL query tool window to help the user avoid these types of issues.

[Matej Mercina]

I find this to be a massive issue honestly. This started to be an issue with debug logs as well, presumably because of the support for increased log size. Just today I ran some anonymous apex code and it coudn’t handle the log file apparently, so it just hung.

When you have a few SOQL/AnonymousApex windows open, and possible other WS instances open for other orgs, this can easily lead to massive amounts of lost work.

There’s one thing i find curious however. Why is it that tools like the Salesforce Inspector browser plugin can handle massive datasets with ease, but WS has issues with only a few thousand rows.

‌

[Scott Wells]

Well, there are two separate issues, one with large SOQL query results and another with the Log Analyzer.

The one with SOQL query results that's the focus of this particular issue is due to the cost of deserialization of the SOAP response into an in-memory result set, specifically when there is either an extremely large number of rows or even a relatively small number of rows but with large BLOB data attached to each row. The only real way around that is to help avoid executing a query that would result in such a response because, once the deserialization of a large payload starts, it would be quite difficult (but not impossible) to terminate it.

The one with the Log Analyzer is because auto-expanding all (or even a large number) of nodes in the tree table is for some odd reason REALLY expensive. This is all part of the plugin SDK. I've debugged and profiled quite a bit, and retrieving and parsing even complex full 20MB logs doesn't take much time at all, but once you put that data into the tree table and tell the tree table to expand itself, it can take a ridiculous amount of time. That's why I've currently limited the number of levels of expansion based on the underlying log size. I might look into writing my own tree auto-expansion implementation to see if I can improve on the current behavior because it just doesn't seem like it should be as expensive as it is.

Anyway, I am planning to address both of these pretty shortly. Hopefully I'll get them both into a state where they either don't ever cause hangs or, if something does run long, you can cancel it.

[Marcel Pękacki]

@Scott Wells Thanks. I don’t know the exact details about how SOQL query functionality works, but I had an idea for a very basic solution: IC could check the size of the SOAP response before the deserialization, and if the response is larger than X bytes (this could be configurable), the IDE would display a confirmation prompt (e.g. “The response is larger than X bytes, its processing could take a very long time, do you want to continue? (Yes/No)”)

[Scott Wells]

Marcel, there's a pretty solid layer in between IC and the raw SOAP response. I use Apache CXF for my SOAP client, and it's responsible for (de)serialization of the SOAP payloads (envelopes, headers, etc.). There are ways to add interceptors for some aspects of the request/response pipeline, though, so I'll investigate whether I can get some insight into the raw response before it's deserialized. If such a provision exists, you're right, I could certainly short-circuit things. I'll investigate.

[Scott Wells]

Delivered in 2.0.8.7. Now IC will warn you when you try to execute an unconstrained SOQL query (no WHERE or LIMIT clause) that includes a Blob-type field or hierarchically-nested sub-queries. You can choose to move forward with the query in which case you'll likely see the same issue, but at least now you'll know that you're in danger of it before the query is issued.