Unit tests on wrong org

[Luke Kozakewycz]

I have a CI tool I’ve built which creates scratch orgs, and updates the source code with the specified branch from git. When I do this - everything in IC works great - except one thing. Unit tests. It seems to stay put on the previous org and is so confusing when you are updating unit tests and nothing changes. When I run the tests from within the org, it works as expected.

It doesn’t matter what I do - it remains on the old org. I can invalidate the cache, restart the machine, etc - nothing works.

Questions:

1. Is it possible to produce a fix that would run the unit tests on the org that IC is assigned to?
2. How can I reset this or change the unit test runner to use the same org?
3. Can I request a small feature to specify which connection is being used (by username rather than or as well as the alias).

Writing these questions makes me wonder if the issue lies with using the alias as opposed to the username when running the unit tests within IC. Just a thought.

Thanks!

‌

[Luke Kozakewycz]

So further to my point I got an error in the unit test runner after I deleted old orgs.

Basically - I had created a new org with the same alias. By doing so, knocked the alias off the old org. After deleting the org I got the following error for which I have no way of working around (yet):

‌

[Scott Wells]

This is actually a bug I've been trying to corner, so perhaps you can help me to do so. Can you provide exact steps to reproduce this behavior consistently? I've tried creating a scratch org with an alias, doing things in IC2 against that scratch org, deleting that scratch org, and creating a new scratch org with the same alias both inside and outside of IC2, and I still haven't been able to reproduce it. Hopefully you have a the magic set of steps that gets me to a repro so that I can fix it.

[Luke Kozakewycz]

I’m going to try my hardest to reproduce without the CI stuff in the way - I’m desperate because I cannot stand using the developer console (who can?). Give me some time and I will get back to you.

‌

[Scott Wells]

Luke, it may well require the CLI's external change, and that's fine if that's the case. If that's the easiest way for you to reproduce this, go that route and just let me know the steps including those that require the CLI. I definitely want to fix this particular bug, though, because it's annoying at best and horribly misleading at worst when you think you've run tests against one org but they've actually run against another...

[Luke Kozakewycz]

Ok so here is what I did:

• Create an org
• Open project in IC
• Run simple unit test
• Create a new org with the same alias
• (The connection will or can be changed in IC)
• Update the unit test to output something different
• Push source and run unit tests

The unit tests will now run on the old org.

This seems to be an issue if using the same alias. When I create a new org with a new alias - it works. And then can easily reproduce doing the same again.

‌

[Scott Wells]

Thanks, Luke. I'll repeat the steps back with a few clarifications. Please let me know if I'm misunderstood along the way, though.

1. Create an scratch org with a specific alias (via the CLI? via IC?).
2. Open project in IC and configure it to use that scratch org.
3. Run simple unit test.
4. Create a new scratch org with the same alias (via the CLI? via IC? with or without removing the original scratch org first?).
   • (The connection will or can be changed in IC) - If it was created with the same alias, IC should use the new connection since it's associated via alias. Was something changed explicitly in this step, though?
5. Update the unit test to output something different.
6. Push source and run unit tests - At this point it either runs against the original scratch org from the first step (if not deleted) or fails (if deleted), correct?

I just want to make sure that I'm doing as close to exactly what you're doing as possible to ensure that I'm able to reproduce the issue.

[Luke Kozakewycz]

1. Create a scratch org with a specific alias via the CLI
2. Open project in IC and configure it to use that scratch org.
3. Run the simple unit test System.assert(false, 'BEFORE');
4. Create a new scratch org with the same alias via the CLI
5. I think IC changes the connection already to the new one
6. Update the unit test to output something new System.assert(false, 'AFTER');
7. Push the source and run unit tests

Correct - at that point I would expect on your end that it would run on the old org or fail if the org has been deleted.

Creating a new org with a different alias seems to produce a different result.

[Luke Kozakewycz]

Note that I’m currently testing the CI process I have in place with a timestamp added to the end to see if that fixes the issue. I will let you know how I get on.

[Scott Wells]

Thanks for the clarifications. And this is something that I will fix as soon as I can reproduce it. I understand why you'd want a workaround in the interim, but if I can reproduce it based on these steps, I'll include a fix in next week's build.

I won't likely get a chance to work on it today, but I'll put it on my work queue for early next week. This one has bugged me (pun semi-intentional) for a while, but without being able to reproduce it it's difficult to guess at a fix...

[Luke Kozakewycz]

It would certainly be a very useful fix to be honest and something that I would happily remove the workaround for.

On that note - the workaround fixed my issue by adding a timestamp onto the alias. The difference is that I was required to change the connection manually - so there might be some methods to modify configurations that would not have occurred when it flipped the connection automatically. Just one of my many thoughts.

Hope I’ve been of some use.

[Scott Wells]

Yeah, I assumed that workaround work...er...work around the issue, and that you would need to nudge IC2 to use the new alias. I'm 99% sure that there's just some cache--presumably persistent since you mentioned that a restart didn't resolve the issue for you--that is still resolving the alias to the original username. Hopefully by being able to reproduce it (fingers crossed that I can), it'll be trivial to track down that cache and ensure that it's evicted/refreshed properly.

[Scott Wells]

Okay, I have a prospective fix for this that will be included in the next build. If you'd like early access to the fix so that you can help to verify it, I can post a pre-release build here for you to try.

[Luke Kozakewycz]

Hi Scott - Yes. I can try it if you haven’t built it already. Is it simple to install/upgrade later?

Thanks

[Scott Wells]

It'll be part of today's official release. If all goes well, I'll have that out within a few hours. You'll get another notification when I resolve this after release. Let me know if things aren't fixed once you update.

[Scott Wells]

Delivered in 2.2.0.2.

[Luke Kozakewycz]

Amazing - thanks Scott! I’ll let you know if I find any issues.