Cannot create module from existing sources

[Mark Mindenhall]

We are in the process of rebuilding our “legacy” Salesforce org (deployments via Copado) as a DX org with CI/CD pipelines. All DX packages in the project are folders within a single git monorepo. Here’s what that looks like from the root level of the project.

$ tree -d -L 1
.
├── academic-evt-consumer
├── benefits-hydration-fetcher
├── canary-evt-consumer
├── catalog-evt-consumer
├── common
├── comms-evt-consumer
├── dev-common
├── dev-common-lib
├── eligibility-fetcher
├── employment-evt-consumer
├── enrollment-evt-consumer
├── fetcher-lib
├── genesis-evt-consumer
├── member-eligibility
├── member-services-app
├── node_modules                    # not a project/package
├── org-setup                       # not a project/package
├── portal-evt-consumer
├── proactive-outreach
├── scripts                         # not a project/package
├── sponsorship-evt-consumer
├── system-admin-app
├── third-party-packages
├── ups-evt-consumer
└── ups-fetcher

‌

When we were kicking off this project back in Q1 of this year, we tested support for this project structure in Illuminated Cloud. We were able to get the following working:

1. Create a new project from existing sources at the repo root. When a bunch of files were detected for Illuminated Cloud, click “Unmark All” to uncheck them and finish creating the project.

2. For each project/package folder in the repo

   1. File → New → Module from existing sources…
   2. Select the folder and click “Open”
   3. Leave “Create module from existing sources” selected, and click “Next”
   4. Now there are “project roots” detected for Illuminated Cloud. Leave these selected and click “Create”
   5. A .iml (module) file is created in the folder, and now the folder in the Project view has the Illuminated Cloud icon on it
   6. We start seeing warnings about things being misconfigured, so we create scratch orgs, etc.

This is an example of an .iml file created in this way:

<?xml version="1.0" encoding="UTF-8"?>
<module type="IlluminatedCloud" version="4">
  <component name="NewModuleRootManager" inherit-compiler-output="true">
    <exclude-output />
    <content url="file://$MODULE_DIR$">
      <sourceFolder url="file://$MODULE_DIR$/config" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/force-app/main/default" isTestSource="false" />
    </content>
    <orderEntry type="inheritedJdk" />
    <orderEntry type="sourceFolder" forTests="false" />
  </component>
</module>

‌

This flow is no longer working with the latest version of Intellij IDEA (2022.1.3) with the latest version of IC (2.2.2.7). When I go through the wizard to create a new module from existing sources, once I click “Create” at the last step, nothing happens. No error is displayed anywhere, and no .iml file is created in the project/package folder.

IMHO support for a monorepo configuration like this is essential. The team members using VSCode are working just fine in this structure, but those of us using IC have been struggling. It would be awesome if this issue could be resolved, and then support for multiple IC modules within a single project can be supported as an ongoing feature.

[Scott Wells]

Mark, this type of project setup should work just fine. Since this is a source format project, I'm assuming it has an sfdx-project.json file in the project root directory, correct? And if so, are each of the desired source root directories explicitly listed as packageDirectories? That's what IC2 primarily uses to determine the default source roots for a newly-created project when an sfdx-project.json file is found.

Let's start there and see what other questions may be required.

[Mark Mindenhall]

Hi Scott! We experimented with the monorepo structure, and for reasons I can’t remember right now (the dev who did the work is on vacation through next week), we are not using a single sfdx-project.json file at the root with a bunch of package folders. Instead, each folder has an sfdx-project.json file. I.e., when we create a new package, we do the following within our monorepo:

sfdx force:project:create -n my-new-package ...
cd my-new-package
sfdx force:package:create --name my-new-package --packageType Unlocked ...
sfdx force:package:version:create --package my-new-package ...

So right now we have 22 folders at the top level of our monorepo that contain sfdx-project.json. It sounds like this structure might be what’s causing the problem for us with IC2? I will say that this structure is working great in all other ways – devs and admins using VSCode are happy, our CI/CD pipelines are awesome, etc. If we can get things figured out with IC2 we’ll be humming along.

Note that we have found ways to work around this problem by manually creating/editing/maintaining the .iml files in each folder, as well as the .idea/modules.xml file.

[Scott Wells]

Okay, gotcha. Yes, what you're describing is best modeled as a multi-module project (assuming that you use IC2 with IntelliJ IDEA and not WebStorm/PyCharm/PhpStorm/RubyMine which don't support multi-module projects). Basically you end up with a single IDE project with one module per-SFDX project that you've created, i.e., one module per-directory with an sfdx-project.json in its root. That's actually how I've set up many of my own projects and it works very well for separation of concerns and modularity.

If you instead prefer (or are required) to maintain things in a single module, you could stub an sfdx-project.json file in the shared project root directory with packageDirectories entries for the various sub-directories and use that for the IC2 project. Obviously not ideal because of the additional scaffolding, but it would also take care of this.

Does that provide what you need? Please let me know if not.

[Mark Mindenhall]

Yes, we are using IC2 with IDEA. But as I described above, setting up the multi-module project is no longer working for us with the latest versions of IDEA and IC2. This was working earlier in the year, so I think there’s a new bug or a regression here.

[Scott Wells]

Mark, I apologize as I feel like I'm spinning you in circles on this based on my misunderstandings. I can see now where you've explained this in your original post. Again, my apologies.

Can you please reproduce the behavior when creating a module where it doesn't complete and then attach your idea.log using Help>Show Log in Explorer/Finder/Files so that I can see what might be happening. I'm not aware of anything that might have changed in this area recently, but hopefully something in the log will help to provide insight into the issue.

[Scott Wells]

FYI, I just did the following:

$ mkdir issue_2145
$ cd issue_2145
$ sfdx force:project:create -d . -n module1
target dir = C:\Users\Scott\dev\projects\IlluminatedCloudTestProjects\issue_2145
   create module1\config\project-scratch-def.json
   create module1\README.md
   create module1\sfdx-project.json
   ...
$ sfdx force:project:create -d . -n module2
target dir = C:\Users\Scott\dev\projects\IlluminatedCloudTestProjects\issue_2145
   create module2\config\project-scratch-def.json
   create module2\README.md
   create module2\sfdx-project.json
   ...

I then used File>Open... in IntelliJ IDEA on directory issue_2145 to create a new project from that directory. It said that it automatically configured the project and gave me the option to take a look at what it had done. I then used File>New>Module from Existing Sources... on module1 and module2 in turn, and now I have a valid multi-module project:

So, the good news is that this should work as you're expecting. Now we just need to figure out why it's not working that way for you. Hopefully your idea.log will have some useful diagnostic information.

[Mark Mindenhall]

Thanks Scott, I will try this again soon. I was out all of last week.

[Scott Wells]

Sounds good. I'm currently traveling but will take a look at anything that you send next week to see if I can figure out why this isn't working properly for you.

[Mark Mindenhall]

@Scott Wells this one can be closed. I have created newer issues for specific problems we see when configuring our project.

[Scott Wells]

Removing component: IntelliJ IDEA (automated comment)