Proposal for $$SEQUENCEFILE$$

[Peter Morse]

Propose to add $$SEQUENCEFILE$$ to return the base file name of the most recently saved or loaded sequence or sequence2. The user could use this value to create a directory that matches the file name to group all the files for the sequence. The question is where should the file name be saved? I’ve considered a static property in class CoreUtil, or, a property in class SequenceSettings. The latter would be saved with the profile and NINA could load the last known sequence upon start up. In that case, should it reload the sequence without asking, ask the user if it should be loaded, or perhaps a new sequence option that tell whether to reload or not? Regardless, of where the file name is saved, I further propose that the current file name be displayed at the bottom of the sequence or sequence2 window in the blank space to the right of the lower left controls. To summarize:

1. Where should the file name be saved?

   1. CoreUtil?
   2. SequenceSettings?
   3. Somewhere else?

2. Should NINA restore the last known sequence?

   1. Automatically?
   2. Ask First?
   3. New Option?

3. Should current sequence file name be displayed at bottom of the sequence window?

   1. Full path?
   2. Just file name with extension?

‌

[Stefan B]

Hi,

the CoreUtil shouldn’t be concerned with a sequencer specific field. That’s not a good place to put it. A profile value could be an option - there are already some for default templates in the simple sequencer and a startup template for the advanced sequencer.

However - I see a problem with the pattern itself - When you load a sequence file and then completely alter the sequence afterwards manually, the pattern doesn’t reflect anything in regards to that file anymore.
What is the use case to have images grouped by a sequence file, rather than the already available image data, like the date, target, filters etc.?

[Peter Morse]

When I do imaging I currently use SGP and I prepare a sequence of targets that I will use over several nights and I want to have all the images grouped together for processing. The fact that I subsequently edit the sequence does not change that need. Grouping by date doesn’t do the job nor is there any other appropriate grouping pattern. In SGP, when you create a sequence you also have to specify a directory for the images. NINA does not require that but there is no current method to gather images based on the sequence. Grouping by date splits the images over more than one directory. Grouping by target doesn’t work because I often will image the same target again in the future and I don’t want them mixed together.

As to being in UtilCore, I chose that because it is common between Image and Sequence. It could easily go into the Sequence profile but there is no need to save and restore it with the profile. I think of the profile as something that reflects the environment and the name of the last loaded sequence does not fit that mission because it is transient.  If you can think of a third place that would be more appropriate, I’m all for it. I considered putting it the Application class but I thought UtilCore was more appropriate.

[Stefan B]

You say grouping by target doesn’t work because the data will be mixed, but you can still just group by target and then by date - something like $$TARGETNAME$$\\$$DATEMINUS12\\<filepattern>

That you saw that the storage of this pattern doesn’t really fit anywhere, is already an indicator that this doesn’t reall fit into the current state of how things work.

I have also asked other contributors and they agree and don’t see why this is required and why a grouping by target name and date is not sufficient… In post processing you will combine images by night and by target (and image type and filter) but never by sequence, so this really doesn’t make much sense to have in our opinion.

‌

[Peter Morse]

Target name and date does not do what I am suggesting. You end up with a series of target directories each of which can have one or more date directories. So when I go to process the images I have to search multiple targets then dates and determine how to combine the images if they where not done on the same date. And what about flat images which I typically only do once for a sequence run. When I process, I’d have to figure out what flats go with what images in multiple directories. Grouping them in a sequence named directory would allow me to see all the images from that sequence in one place. As I explained, I’m used to SGP where you have a single directory (and subdirectories) to hold all your images from a sequence. When I looked at NINA it was clear that there was no good way to consolidate all the images from one sequence. Having many years of development experience, I certainly understand the concept of not adding random features to NINA, but this feature has a definite purpose and is unobtrusive to users who don’t wish to use it. I’m suggesting this feature as a current SGP user who is considering using NINA but I am finding that it has some annoying limitations and this is one of them. I have other examples of where I think NINA falls short but if I can’t convince you of this minor feature, it seems I am wasting my time and effort to contribute. When developing software, it is easy to go for the grand features and overlook the little details that really improve usability. This is such an example.

[Linwood Ferguson]

I am still trying to work out my own post processing workflow for multi-night imaging so was following along in this as a potentially helpful step, but find I am more confused…. would it be possible for you to describe the end result? What folder structure you would like after all is said and done? And why?

As a for example – I file now by target (implying framing/rotation) and subfolder by filter (and I guess focal length, though there’s rarely overlap). After calibration. That’s all I need, I can’t see why I need to know anything about the sequence that produced it. Do I?

But maybe I’m missing something.

[Peter Morse]

For testing I used this pattern:

\$$SEQUENCEFILE$$\$$IMAGETYPE$$\$$TARGETNAME$$-$$FRAMENR$$

So I end up with all my targets for a sequence grouped by image type in a common directory. I think $$SEQUENCEFILE$$ would generally be used as the first thing in a pattern and then whatever you want after that. The reason I like this organization is that I name my sequence files with the current date as part of the name. With SGP, I then use the corresponding date name for the images directory. Even though I may image over several nights, the sequence file name and image directory name have the beginning date in them. I’ve been doing this for years and I have all my sessions organized by date. I have two big drives so I can keep many years of data although I rarely go back to them. When I finish a multi-night imaging session, all my images for the session are grouped together and I can find them easily for processing and I also put my processing directories in the same date named directory so everything is all together. With NINA, you can have the date as part of the pattern but for a multi-night session you end up with different dates that are all part of the same session. So, I hope this explains why $$SEQUENCEFILE$$ is a useful pattern for me and hopefully for others.

‌

[Dale Ghent]

Hi Peter,

It seems that you’re used to operating in a way that uses a certain feature or behavior in SGPro and are looking for an analog of that in NINA. I have to say that this is the first time anyone that we know has popped up with this specific kind of requirement and use case so, naturally, there is a bit of questioning around whether this is a change that would be of benefit to the wider userbase or if it is something that scratches one person’s very particular itch. As you might know, it’s hard to justify a change on whether it might be useful to some unknown population. Also, this change touches areas that aren’t normally touched in terms of image metadata so this brings some extra scrutiny on top of that.

Normally, we do try not to be a facsimile of other apps or cargo cult in unique features or behaviors because there’s generally a wish to avoid being perceived as a copy of, or copying, other apps. So reasons like “because SGPro has/does it” reflexively prompts questions such as has the user has considered either changing the way they do things or attaining their goals through existing altough different means, and I think that’s what has transpired here.

That said I support this concept but only if it’s implemented in a way that’s in keeping with the established code environment - ie, it’s not so outrageous of a change that it sticks out like a sore and unique thumb. The final call is with Stefan, of course.

It’s also really encouraged that contributors be a part of the community so that they can help support users of the code they contribute, so please feel free to join the NINA Discord chat and hang out

[Linwood Ferguson]

Peter, thanks for the explanation, I have to think about it a bit. I have no opinion on the change requested.

One note, when you say “With NINA you can have the date as part of the session”… “different dates”, do you know about the $$DATEMINUS12$$ feature, it keeps all for one night together (the date when it starts) and doesn’t split the data at midnight.

Personally I am moving more toward separation of calibration and other post processing. Assuming no rotation change, I take the entire night’s runs and do calibration on them the next morning, regardless of target. I then file them by target and filter, as date is then irrelevant (I discard the originals and keep only the calibrated frames). This simplifies multi-night a lot. But is probably off on a tangent from your request.

Thanks for allowing the discussion.

[Peter Morse]

Dale, certainly agree with your comments. I’m not suggesting NINA needs to duplicate SGP but it is fair to consider similar usages. SGP requires you to specify a directory for a sequence. My suggestion for NINA adds a new image pattern to be able to do a similar thing but following the NINA methods. This is only a few lines of code and adds one image pattern which anyone is free to ignore - hardly going to break NINA.

Linwood, yes, I’m aware of $$DATEMINUS12$$, but it does not solve the basic problem of having multiple dates for a single multi-night session.

Everyone, my current alternative is to go into NINA and change the image directory options every time I create a new sequence so that my images are saved in a new place. Needless to say not a great “feature”. $$SEQUENCEFILE$$ would allow me or anyone to automate the process. It’s a tiny change which is very useful for those that care about it. There are a lot of image patterns in NINA that I don’t need so I just ignore them. So it would be for $$SEQUENCEFILE$$ for other users.

[George Hilios]

As another contributor and heavy user I thought I’d chime in as well - I’m not a fan of this proposal as it is defined. The primary reason is the use of a file name as a keyword. The filename itself is nothing special, and so changing the file name to produce a different output path is awkward at best and limits flexibility in the future. I also think it would be confusing.

If we do anything like this at all, I suggest we make a variable that someone in the sequence can edit directly. Like a “Sequence Name” or something, and expose it as a keyword that can be added to the file metadata or used in the output path.

Even then, I suspect this won’t be a commonly used feature as no one has asked for it or lamented it missing. In the spirit of limiting bloat to the overall application, I don’t think we should make this change until or unless there are several users asking for it.

[Peter Morse]

The file name is a natural sequence name that already exists. Adding another level of indirection with a sequence name variable adds no additional clarity. As to bloat, it’s a couple of lines of code and one keyword, and as a side benefit, load and save sequence remember the last used file name in the dialogs.

Your comments do not address the issue I was trying to solve. If a user images over several night using the same sequence, how do they aggregate the results in one place? I’m sure I am not the only person who images over more than one night who wants to have their results in one location. None of the current keywords addresses that need. Do you have an alternative proposal that would address this common scenario?

[George Hilios]

I image the same target over several nights, and I group by target name + date. There are multiple ways to solve this problem, and the NINA community seems to have gotten behind a model that doesn’t involve using a sequence name to group. I’m aware of several other members on Discord that do something similar.

The problem isn’t so much about “lines of code” (I agree, this is a simple change), but rather - complicating the user experience by adding more knobs, bells, and whistles. This type of thing becomes a slippery slope quickly, so we (the contributors) do our best to remain vigilant. We often have our own ideas rejected by the other contributors, in fact. My recommendation is to see if there’s demand among the community for solving this problem in the way you propose. Otherwise we risk adding some functionality that everyone sees but only 1 (or very few) people use.

[George Hilios]

To add one more point regarding using the file name - You don’t have a file name when you are first creating the sequence in the UI. I see that you handled this in the code, but the result is somewhat non-deterministic behavior. You might get one value for the sequence name when creating it with the UI, but then something else when you save and reload it later. Even if we support sequence names as a keyword, I am fairly strongly against using file name to do so.

[Peter Morse]

I don’t think you are addressing the scenario I am suggesting. I’m not sure I understand your point about sequence file names. It’s very unlikely that a user will create a sequence and then not save it before beginning imaging so the keyword value will be known. Most of the time I’m going to choose a file name and that is that. Of course, I could rename the file outside of NINA but I’ll know that. There is nothing inherently confusing about using a file name as a grouping method as the user has control of the process. Like other image keywords $$SEQUENCEFILE$$ has a default value of ““.

What you are suggesting breaks down the image by target and date. When I go to process a set of images from a run over several nights, I want to find all the target images in one location. When I process in PixInsight (or whatever), I select all the images for a target to do a blink comparison. I don’t want the images split between date directories because they where done on different nights. Also, it is likely I will do Flats on yet a different night. So with your method, I have to dig around in date named directories to determine my sets of images and flats. I guess I could locate all the images and move them to a common directory before starting processing. Or, I could forgo image patterns altogether and just used a fixed directory name. But, I’d rather automate the directory selection. Of course, automation is the whole point of NINA so let’s support that. An extra image pattern of $$SEQUENCEFILE$$ is not going to confuse anyone.

The objection to using the file name as a grouping method is a bit of a red herring. It’s a natural way to group multiple nights together. I can set up a directory pattern with $$SEQUENCEFILE$$ and use it for years since NINA will handle it for me rather than having to remember to set up a specialized directory each time. Using target and date does not group the images by the sequence file. Users have been doing it the way you suggest because there is no good alternative. Once this feature is available I won’t be the only person using it.. We don’t have to wait for users to think of the possibility of a feature on their own before implementing it.

‌

[George Hilios]

It’s ultimately up to our benevolent dictator, but my 2 cents is that I’d like to see:

1. More user interest in this capability/approach to solving this problem
2. The file name no longer being a part of this

You mention that it’s “highly unlikely” that a user would start a sequence before saving it, but I personally don’t have to look far (I do this frequently, since I use templates to create new sequences frequently). Furthermore, it’s not clear to me why having a sequence name grouping acquisitions together accomplishes more than viewing a directory structure partitioned by target and date. Our users have been doing this to-date, and you’re the first to suggest it’s a barrier to productivity.

The real trigger for me here is the use of the file name and the non-determinism it introduces. If you found a natural alternative to this that provided the same, deterministic benefits for a pure UI-based use case, then my criticism essentially fizzles away to “I wouldn’t personally do it that way, but to each their own”.

I may be wrong about this point, but my impression on this is that you’ve gotten used to a workflow by working with SGP, and you want to keep as close to the same workflow in NINA. I understand the desire to do that, but I also encourage you to think about whether you’re providing a better way of addressing an existing/new problem, or simply providing an alternative. This smells like the latter, and if so - I believe we should hear from more users asking for this before moving forward.

[Peter Morse]

Although it is a workflow I use in SGP, it is not SGP specific. SGP requires you to use a base directory with each sequence file and then the file patterns are based off of that. So NINA is actually more powerful in that you can automate the complete directory path - not just part of it. Without the subject feature, I would have to do as I do in SGP and enter a new directory with each sequence file. The point of $$SEQUENCEFILE$$ is to allow complete automation of the file path. I’m not sure why you feel using a file name is non-deterministic. There is nothing magic about the file name - the user certainly knows what it is.

As a matter of fact, I’ve been looking into displaying the base file name with the sequence which would make the file name and keyword more apparent. On the current advanced sequence form, it displays “Sequence” as a title which is not very informative. In test code I have, it now displays either “Sequence” or the base file name if it has been saved or loaded. Would that make you find this feature more deterministic? What would you think of having $$SEQUENCEFILE$$ return “Sequence” when the sequence has not been saved? That would create a situation where $$SEQUENCEFILE$$ would always matched the displayed title.

I would like to do the same for the target set screen, however, there is no obvious place to display the file name since it does not have a title but I’ll see if I can find a place to display the filename.

[Stefan B]

Having the title as a pattern would make a lot more sense than having the file name. This would allow for people that do not save their sequence to also make use of this pattern. With the advanced sequencer you don’t have to save your sequence, when most of your planning is done using the templates and target system instead.

So instead of sequence file we could go with $$SEQUENCETITLE$$ which is a lot more intuitive. A title to be exposed to the simple sequence UI shouldn’t be much of a problem either.
Having the title to auto adjust after save also seems reasonable.

[Peter Morse]

Ok, let’s clarify a few details.

So $$SEQUENCETITLE$$ would have to know whether a simple sequence or advanced sequence was running and choose the appropriate title?

In either sequencer, the title would be either a default name like “Sequence” or the base name of the last saved or loaded file?

In the case of the simple sequencer, the title would be the target set file name if any and not the name of an individual target?

Any thoughts where on the simple sequence form you would like to see a title field? An appropriate place might be above the target name “arrows” but real estate is tight on that form. That location would be consistent with the advanced sequencer.

When I did a trial of this on the advanced sequencer, I wondered what the default title should be. “Sequence” seems like it could be easily confused with a file name. Maybe the simple sequencer should have a different default title than the advanced sequencer? (“Target Set” and “Advanced Sequence”) Whatever is used it needs to be a valid path component name.

I like the overall approach of this proposal. It addresses the multi-night scenario and it provides file name feedback in the UI.

‌

[Stefan B]

the macro doesn’t need to know if it is an advanced or simple seuqence. The one that is running drives the name on image saving, similar to how a target name is determined.

For the location inside the simple sequencer, i would need to try out a few options. A separate row on top could make sense.

I’m open to changing the default names of both to something more meaningful. The initial value is more of a placeholder anyways.

[Jerry Macon]

Peter’s initial statement: “In SGP, when you create a sequence you also have to specify a directory for the images. NINA does not require that but there is no current method to gather images based on the sequence.“

is not really true. NINA does require you specify a directory for the images.

Why does this not exactly duplicate what he is doing with SGP?

Just the fact that it is not stored with the sequence? If this is such a big issue, just change it there when you start a new sequence.

However, adding the sequencetitle seems like it could be an improvement that would not complicate the program and not hard to add.

[Peter Morse]

There is a fine point here. In SGP you must define a directory for EACH sequence. You can’t start a SPG sequence without a directory defined. In NINA, you define a directory in setup which applies to ALL sequences. So in NINA, if you set up the base directory and the right image patterns, you don’t have to remember to set up a directory when creating a new sequence. Of course, you can change the base directory in NINA each time you run a sequence but why would you want to to that if it can be completely automated.

[Jerry Macon]

Yes, if that is the workflow that you want to continue using, this feature (may) save you that one step once you start a new sequence. Of course when you build a new sequence, you must put in the new path then also, so the actual effort is exactly the same: when you build a new sequence, you must put in a new path SOMEWHERE. But clearly you are able to continue your SGP approach with the current NINA, you just put the path in a different place when you start a new sequence.

OR, you could try a slightly different approach to the workflow.

1. always download new subs to the SAME directory. But within that directory there is a separate directory titled by NAMEofTARGET for each target.
2. Next morning: (optional, but what I always do) Purge bad subs from the collection
3. Transfer these files to your sequence named directory, or to the individual TARGET directories, which is the useful thing to do.

Since collecting subs together by TARGET is what is really important, I don’t see any benefit to collecting them by sequence. In fact, more often than not, I have a different mix of targets on different nights. It is rare to be able to image the same target all night long, so generally you have more than one target. I usually finish those targets on different nights. Different targets have different requirements, particulary how much integration time is needed to get good results. For this reason, collecting subs by sequence would be a disaster.

‌

[Peter Morse]

Of course, you use target as a directory level below sequence so it’s not a disaster. All the target images are together as you would expect. If it doesn’t work for you then don’t use it. I’ve used the approach for years with great success.

[Peter Morse]

Implemented $$SEQUENCETITLE$$ per these discussions on original feature/$$SEQUENCEFILE$$ branch (pending pull request). Please review and if OK I’ll add unit tests and release note comments.

[Stefan B]

$$SEQUENCETITLE$$ is now merged. Thank you for the contribution.

@Peter you haven't yet put your name into the contributors file. Please feel free to add it there if you want to be mentioned :)

[Stefan B]

Removing version: 1.11 Nightly (automated comment)