- attached projectdns_toveostest_07171335_.err
- attached projectdns_toveostest_07171335_.out
keep running into nan ,need help
Issue #2621
closed
I try to run a double neutron star sim,using a cold eos table,but i keep running into nan for months,the par file and the output and err file are in the attachments.please give me some hints on where could be going wrong,I really have no clue.
there are other sim i used,but the result are the same,usually it started normal,everything looks fine,the center density the stable,the rotation looks ok,but later on,just when the matter from two stars started to interact with each other,something always happen,either
nan on the start of the interaction,or nan in the process of merge. At first i thought maybe it became a blackhole,but the i checked the center density,it did not show signs of continue increase,often it just running normally and nan just happened.
i can provide the eos table,or anything you need,please,i just need some hint,i tried everything i know,i am drying out here
Comments (16)
-
reporter
-
reporter
- edited description
-
reporter
- edited description
-
reporter
- edited description
-
reporter
I also tried using FMR instead of AMR,it just keep running into nan:
INFO (NSTracker): Found star at (-5.90625,7.03125,0) 95572 1343.981 | 59.3475136 | 0.0000945 0.0020815 | 0.0008303 | 1.6975849 | 0.0068918 | 2402 INFO (NSTracker): Found star at (-5.90625,7.03125,0) 95576 1344.038 | 59.3481466 | 0.0000945 0.0020816 | 0.0008302 | 1.6958857 | 0.0068909 | 2390 INFO (NSTracker): Found star at (-5.90625,7.03125,0) 95580 1344.094 | 59.3485846 | 0.0000945 0.0020817 | 0.0008302 | 1.6936211 | 0.0068890 | 2395 INFO (NSTracker): Found star at (-5.90625,7.03125,0) 95584 1344.150 | 59.3489886 | 0.0000945 0.0020818 | 0.0008302 | 1.6895956 | 0.0068864 | 2390 INFO (NSTracker): Found star at (-5.90625,7.03125,0) 95588 1344.206 | 59.3487586 | 0.0000945 0.0020819 | 0.0008302 | 1.6834992 | 0.0068836 | 2395 INFO (NSTracker): Found star at (-5.90625,7.03125,0) 95592 1344.263 | 59.3493541 | 0.0000945 0.0020819 | 0.0008302 | 1.6749728 | 0.0068800 | 2390 INFO (NSTracker): Found star at (-5.90625,7.03125,0) 95596 1344.319 | 59.3496240 | 0.0000945 0.0020820 | 0.0008302 | 1.6658542 | 0.0068756 | 2395 INFO (NSTracker): Found star at (-5.90625,7.03125,0) ----------------------------------------------------------------------------------------------------------------------- Iteration Time | *me_per_hour | GRHYDRO::dens | *ROBASE::rho | *::w_lorentz | *STRAINTS::H | *axrss_mb | | sum maximum | maximum | maximum | maximum | maximum ----------------------------------------------------------------------------------------------------------------------- 95600 1344.375 | 59.3502048 | 0.0000945 0.0020820 | 0.0008302 | 1.6612792 | 0.0068712 | 2390 INFO (NSTracker): Found star at (-5.90625,7.03125,0) 95604 1344.431 | 59.3502108 | 0.0000945 0.0020820 | 0.0008301 | 1.6612805 | 0.0068671 | 2402 INFO (NSTracker): Found star at (-5.90625,6.75,0) 95608 1344.488 | 59.3509198 | 0.0000945 0.0020820 | 0.0008301 | 1.6559333 | 0.0068783 | 2390 INFO (NSTracker): Found star at (-25.6585,-0.0634766,25.6993) 95612 1344.544 | 59.3509957 | -nan 1.552986e-11 | 1.000000e-11 | 1.0000000 | 0.0008295 | 2395
-
reporter
- attached dns_AMR.par
</div> </form>
-
reporter
- attached AMR.out
-
Hello Artectek, this was discussed in today’s public ET call, and some notes are in the minutes: http://lists.einsteintoolkit.org/pipermail/users/2022-July/008619.html
-
reporter
@Roland Haas ,thanks a lot,but i see there are some suggestions regarding this problem? may I know what are them?
-
Two items were brought up:
- the resolution may well be too low. The gallery example (which your parfile dns.par seems to be base on) sets the resolution to a very coarse value, namely 18Msun on the coarsest level and 18Msun/2^6 = 0.28 for the level that covers the star(s) initially. This corresponds (using Msun ~~ 1.48km) to a resolution of 416m which is too coarse for any serious simulation. It should be no more than about 280m so all resolutions should be scaled to dx=dy=dz=12
- Looking at your log output then at t=0 there is: “INFO (CarpetTracker): Setting position of refined region
#1from surface #0 to (13.5,0,0)” indicating that the star was found with center at x=13.5 however you initial grid structure for the refined regions (the CarpetRegrid2::position_x parameters) centers the boxes around x=22.5 which is not correct (probably not the cause for the failure at the end but incorrect and will lead to an initially incorrect grid structure). You can open eg 2D output files (hydrobase-rho.xy.h5) in VisIt and add a “Mesh” plot to check what things look like at iteration=0.
-
- marked as task
-
reporter
thanks,I will try that right away
-
reporter
@@Roland Haas ，sorry to bother you again, but i tried your advice,setting dx=dy=dz=12,it still ran into nan at about the same time as before
INFO (NSTracker): Found star at (9.375,5.625,0) 139444 1307.287 | 61.5954977 | 0.0003189 0.0020122 | 0.0008347 | 1.4536858 | 0.0038059 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139448 1307.325 | 61.5964119 | 0.0003189 0.0020122 | 0.0008347 | 1.4639129 | 0.0038125 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139452 1307.362 | 61.5969842 | 0.0003189 0.0020122 | 0.0008347 | 1.4745400 | 0.0038180 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139456 1307.400 | 61.5976158 | 0.0003189 0.0020123 | 0.0008347 | 1.4825708 | 0.0038222 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139460 1307.438 | 61.5972051 | 0.0003189 0.0020123 | 0.0008347 | 1.4907403 | 0.0038258 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139464 1307.475 | 61.5980420 | 0.0003189 0.0020124 | 0.0008347 | 1.4315498 | 0.0038305 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139468 1307.513 | 61.5987091 | 0.0003189 0.0020124 | 0.0008347 | 1.4378927 | 0.0038341 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139472 1307.550 | 61.5993935 | 0.0003189 0.0020125 | 0.0008348 | 1.4155038 | 0.0038366 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139476 1307.588 | 61.5998317 | 0.0003189 0.0020125 | 0.0008348 | 1.4033181 | 0.0038379 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139480 1307.625 | 61.6007500 | 0.0003189 0.0020125 | 0.0008348 | 1.4024480 | 0.0038378 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139484 1307.662 | 61.6013809 | 0.0003189 0.0020126 | 0.0008348 | 1.4011666 | 0.0038363 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139488 1307.700 | 61.6020977 | 0.0003189 0.0020126 | 0.0008348 | 1.4000632 | 0.0038334 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139492 1307.737 | 61.6018872 | 0.0003189 0.0020126 | 0.0008348 | 1.3990310 | 0.0038290 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139496 1307.775 | 61.6028024 | 0.0003189 0.0020126 | 0.0008348 | 1.3982709 | 0.0038232 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,5.625,0) 139500 1307.812 | 61.6033686 | 0.0003189 0.0020126 | 0.0008348 | 1.2511643 | 0.0038163 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,5.625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-5.625,0) INFO (NSTracker): Found star at (9.375,10.875,0) 139504 1307.850 | 61.6041180 | -nan 0.0014981 | 0.0006712 | 1.0485346 | 0.0019915 | 2345 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (9.375,10.875,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-9.375,-10.875,0) INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1 WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=5 c=0] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty()
-
Hmm, I cannot say I have a good suggestion how to proceed, I am afraid. If this is just around the time that the stars come into contact then it could be a con2prim failure which is very hard to debug, in particular since I do not think that the cold tabulated EOS has been heavily used for binary simulations.
-
reporter
To my understanding ,cold tabulated EOS is just a multipoly eos with much more points,i tried increase the resolution even more to ~140m,but it is still the same,this time is:
INFO (NSTracker): Found star at (11.5312,-2.90625,0) 138090 1294.594 | 6.1129402 | 0.0003189 0.0019738 | 0.0008331 | 2.0279669 | 0.0010922 | 19715 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (11.5312,-2.90625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-11.5312,2.90625,0) INFO (NSTracker): Found star at (11.5312,-2.90625,0) 138092 1294.612 | 6.1129794 | 0.0003189 0.0019739 | 0.0008331 | 2.1415209 | 0.0010950 | 19715 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (11.5312,-2.90625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-11.5312,2.90625,0) INFO (NSTracker): Found star at (11.5312,-2.90625,0) 138094 1294.631 | 6.1129793 | 0.0003189 0.0019739 | 0.0008331 | 2.2870158 | 0.0010950 | 19715 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (11.5312,-2.90625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-11.5312,2.90625,0) INFO (NSTracker): Found star at (11.5312,-2.90625,0) 138096 1294.650 | 6.1130046 | 0.0003189 0.0019739 | 0.0008331 | 1.8727232 | 0.0010921 | 19715 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (11.5312,-2.90625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-11.5312,2.90625,0) INFO (NSTracker): Found star at (11.5312,-2.90625,0) 138098 1294.669 | 6.1129553 | -nan 0.0019739 | 0.0008331 | 1.2863624 | 0.0010853 | 19715
-
- changed status to closed
closed due to lack of activity
- Log in to comment
- Assignee
- –
- Type
- Priority
- Status
- closed
- Component
- EinsteinToolkit thorn
- Milestone
- ET_2022_05
- Version
- ET_2022_05
- Votes
- 1
- Watchers
- 1
