Single project, multiple repos, multiple packages (2GP only)

[Attila Hajdrik]

Disclaimer: I’m not intellij ninja.

Scott, I’m looking for guidance on the following:

We’ve a project setup with multiple 2GP packages, each package lives in their own Git Repository. Packages are dependent on each other and have all the goodies a project has. All the packages are in the same namespace, which is identical to the namespace of our DevHub.

Here is the hierarchy:

• Core

  • No deps

• Common

  • Deps: Core

• Feature1

  • Deps: Core, Common

• Feature2

  • Deps: Core, Common, Feature1

• Feature3

  • Deps: Core, Common, Feature2

‌

What I’d like to achieve for starter is:

• 1 IntelliJ project
• Push and Pull to same Scratch org during development
• Setup connection(s?)
• Setup modules
• Setup substitution rules if needed
• Make sure source push-pull works
• Setup .gitignore in regards to IlluminatedCloud requirements on what should go in and what not
• Feature1 Defines MyObject1__c and Feature2 adding fields to that object, so source/push-pull somehow must be able to have this configured. No idea where would a new field added to MyObject1__c would be pulled, but if it is pulled, could be moved later.

‌

I think that this scenario will become more common between ISVs and it would be quite useful to have this documented how to achieve this.

[Scott Wells]

Hi. Generally you're correct that you'd want to use modules for this where each node in the hierarchy would be its own module, and you would set up module dependencies pretty much as described. All modules would use the same connection, and if that connection has a namespace, that would also be used.

However, because you're describing this to be against a source-tracked scratch org, that's unlikely what you're going to want. Instead you'll likely want a single-module project mapped directly to the Salesforce DX project, and in sfdx-project.json you'll have multiple packageDirectories entries, one per-"module" root. That way Salesforce DX's source tracking mechanisms will work properly during push/pull actions.

Note that substitution rules do not work with push/pull, only with deploy/retrieve. If you're working against a source-tracked org using push/pull, that won't be an available feature.

There shouldn't be any issues with using a .gitignore. That's all handled by IntelliJ IDEA. I've documented IDE config files in version control and would recommend checking them in for use by the entire team.

As for how pull works when you have multiple package directories and pull new metadata from the org into the project, you must designate exactly one of your package directories as "default": "true", and metadata will be pulled into that package directory when it can't be matched against an existing local counterpart. You can then, of course, move those files into the appropriate package directory using the IDE and the next push should realign things in source tracking.

Let me pause there and see what additional questions you have...

[Attila Hajdrik]

Thanks Scott for the - as always - valuable insights. I was thinking about mono repo and leverage packageDirectories as it seems to be working better now than it was working before, when SFDX and 2GP was just new, but as the product grows and the number of managed packages are growing as well, it could get really easy out of hands, issue tracking, releases, etc, I don’t think I would go down that route, that’s why I’m looking for a solution with multiple modules and multiple Git repos.

"default": true works until it is not, if you have the MyObject__c in /feature1/objects than when you add a field, IIRC source pull is "smart" enough to put the new fields into /feature1/objects/MyObject__c/fields and not into the default one.

On the “monorepo” route, what do you think of git subtrees, better than submodules, and probably could work in concert with packageDirectories ?

My thoughts are: the “main” repository would be the one that has everything in the root, sfdx-project.json and such, this would be the default, and every other repo would be in a directory at the root, mapped to an entry in packageDirectories.

Any other ideas how could it work efficiently?

[Scott Wells]

If you want my honest take, don't use push/pull. IC2 was designed from its inception to handle multi-module projects with cross-module dependencies, but in order to do so properly, it has to manage the movement of metadata back and forth between the project and the org. If you're not totally married to using source tracking for these projects, I'll be happy to explain in (likely excruciating) detail how this would work best in IC2.

[Attila Hajdrik]

Please do I just started to use push/pull because … I thought that is the future of 2GP, if you like to share some links to read before going deep into the topic just share it I’m happy to do the homework before!

[Scott Wells]

I'll be happy to provide a brain dump, but it'll likely be early next week before I can do so. Wrapping up a family spring break getaway right now with travel tomorrow followed by the weekend. I'll flag this for a response on Monday and, once I dig myself out of the beginning-of-week stuff, I'll put together a solid overview for you.

[Attila Hajdrik]

No rush, enjoy the travel, family first
I hope that this topic will be useful for others as well and could be compiled into a doc page or a video even?! who knows!

[Scott Wells]

Okay, let me ask a few clarifying questions so that I can provide more accurate guidance:

1. If I understand correctly, in the expressed hierarchy each module/node (e.g., "Core" vs. "Common" vs "Feature1/2/3") has its own Git repo.
2. All modules will be deployed as source into a single org. In other words, you're not installing "Core" and "Common" into an org as packages, then deploying "Feature1/2/3" as source.
3. You mentioned "Setup substitution rules if needed". Are there specific use cases for substitution rules that you're anticipating, or was that more of a "cover the bases" type of comment?

Let's start with those and hopefully with the answers to those questions in hand, I'll be able to provide a detailed response for you.

[Attila Hajdrik]

During the weekend, I rewatched all the videos of IC to get more sense, and played with a single project with deploy/retrieve based on subscription and I loved it. Much better then the “shelled out” sfdx source push….where sometimes no error was returned but it failed…so I would say I have some grasp of its operation.

Regarding your questions:
1. Yes you are correct and each of these are a managed package.

2. Here comes the weird thing when we first tried source deployment I was not able to install 2 package and source push the 3rd one which had the same namespace (we talked about this back back in the day in email)
3. Cover the bases type of comment, if not needed then it is just better.

‌

Sorry for the delayed answer it is hell of a busy week.

[Scott Wells]

FYI, I'm still planning to provide a solid response to this, but things have been pretty busy over the past week and will be for the next few days. Rest assured, though, that it's sitting at the top of my "breach the surface" queue!

[Scott Wells]

At the risk of being a skipping record, I know I've been horribly unresponsive on this. Things have continued to be insanely busy. I'm hoping once I get through this week I'll have time to provide a proper response. I'm keeping this at the top of my TODO list so it's constantly nagging me, so a response will come...it's just a matter of time (and hopefully a short matter of time at that).

[Mateusz Michalczyk]

Hi Scott, Attila,

I have recently started to use IC and it’s been great so far but recently a project has been changed so that we have now multiple folders defined in sfdx-project.json the development is against a Sandbox, not a scratch org.

Since then IC does not respond to retrieve or save on the files in the defined folders. Also there are plenty errors as the offline symbol table is obviously not pointing to these folders. I have quickly opened that repo in VS Code and retrieve and save had no issues. I’m not sure what I am missing to make it work.

I feel like my issues is similar to the package development issue although no packages are involved on my side, just separate folders.

Some part of sfdx-project.json

"packageDirectories": [
{
"path": "./force-app/b2b-commerce",
"default": true
},
{
"path": "./force-app/b2b-commerce/common",
"default": false
},
{
"path": "./force-app/b2b-commerce/mynexeo",
"default": false
},

‌

‌

in my iml file I have this although when I was creating the project I did not tick “ Generate Manifest”
<ModuleContents> <option name="contentSelectionType" value="PACKAGE_XML" /> <option name="packageXmlRelativePath" value="manifest/package.xml" /></ModuleContents>

‌

Any ideas ?

[Mateusz Michalczyk]

eh, nvm solved by changing : <sourceFolder url="file://$MODULE_DIR$/force-app/main/default" isTestSource="false" />

into <sourceFolder url="file://$MODULE_DIR$/force-app" isTestSource="false" />

‌

[Attila Hajdrik]

Scott, have you thought about the solid response?

[Scott Wells]

Hi, Attila. I'm so sorry for the lack of response. For some reason I thought that Mateusz' "nvm solved ..." response above signaled that you and your team were in good shape, so I just filed this away. Sounds like that's not the case and you're still wanting a description of how to set up a multi-module project for dev in a single org, correct? And these would all be source format modules, but you're fine working against the org using deploy/retrieve/delete instead of push/pull? If that's the case, what I may do is just put together a skeletal multi-module project for you that models this and even includes a few simple substitution rules for reference. Please confirm whether what I've said here is correct and I'll get that over to you to see the next round of discussion.

[Scott Wells]

Issue tracker grooming. If this is still an issue, please feel free to reopen.