multiprocessing not working for me
Issue #22
closed
Using multiprocessing (2 processes) will not work.
It opens a second QGIS at a certain point, where the following errors are reported:
2020-06-24T13:46:39 CRITICAL Invalid Data Source : C:\Users\Fabian\Documents\-c is not a valid or recognized data source.
2020-06-24T13:46:39 CRITICAL Invalid Data Source : C:\Users\Fabian\Documents\from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=12892, pipe_handle=7092) is not a valid or recognized data source.
2020-06-24T13:46:39 CRITICAL Invalid Data Source : C:\Users\Fabian\Documents\--multiprocessing-fork is not a valid or recognized data source.
Log:
Started at 2020-06-24 13:46:26.628759
Parameters(root='C:\\Users\\Fabian\\Downloads\\temp\\level2', outputRoot='C:\\Users\\Fabian\\Downloads\\temp\\output', outputExtension='.tif', sensors=('LND04', 'LND05', 'LND07', 'LND08'), roi='C:/Users/Fabian/Downloads/temp/roi.shp', tilenames=None, prefereLandsatImproved=False, dateRange=None, dateSeason=None, qaiFlags=(<Qai.ValidNo: 2>, <Qai.CloudLessConfident: 4>, <Qai.CloudConfident: 5>, <Qai.CloudCirrus: 6>, <Qai.CloudShadowYes: 8>, <Qai.SubzeroYes: 18>), features=(<Feature.Blue: 1>, <Feature.Green: 2>, <Feature.Red: 3>, <Feature.Nir: 8>, <Feature.Swir1: 9>, <Feature.Swir2: 10>), products=(<Product.Median: 3>,), percentiles=(0, 5, 25, 50, 75, 95, 100), timeseries=False, binning=<BinningOption.No: 2>, binStart=DayOfYear(month=1, day=1), binDays=10, bins=None, resolution=Resolution(x=30.0, y=30.0), resamplingBoa=<Resampling.Average: 'average'>, resamplingQai=<Resampling.Mode: 'mode'>, relativeToVRT=False, open=<OpenOption.Mosaic: 1>, processes=2, blocksize=GridShape(y=100, x=15000))
Parse Level 2 folder
Parse X0068_Y0038
Parse X0069_Y0038
Parse X0070_Y0038
Prepare QAI masking
RasterCollection.map(<function mask at 0x0000015E10349488>) 0%..2%..5%..7%..9%..11%..14%..16%..18%..20%..23%..25%..27%..30%..32%..34%..36%..39%..41%..43%..45%..48%..50%..52%..55%..57%..59%..61%..64%..66%..68%..70%..73%..75%..77%..80%..82%..84%..86%..89%..91%..93%..95%..98%..100%..Done
Prepare features
Exclude unwanted wavebands
RasterCollection.map(<function Raster.select at 0x0000015E1047B840>) 0%..2%..5%..7%..9%..11%..14%..16%..18%..20%..23%..25%..27%..30%..32%..34%..36%..39%..41%..43%..45%..48%..50%..52%..55%..57%..59%..61%..64%..66%..68%..70%..73%..75%..77%..80%..82%..84%..86%..89%..91%..93%..95%..98%..100%..Done
Prepare composites
RasterCollection.reduce(<function medianComposite at 0x0000015E10362AE8>) ..Done
Also, GUI says “Done!” but the action button is deactivated. So in order to rerun the plugin, one has to restart QGIS. Task seems to continue running in the background with 100% progress.
I ran a test on pretty much the default settings with a 3 tile force level2 output. Single process run works.
Comments (6)
-
-
Why did it fail?
-
@Benjamin Jakimow try to run this (multiprocessing) code in your QGIS Python Console. QGIS will freeze and new QGIS instances will pop up:
import multiprocessing multiprocessing.Pool().map(print, [1,2])
-
- changed status to closed
-
It is at least running in another thread to avoid blocking the QGIS GUI?
-
Sure, back then we decided to use QgsTask to at least achive concurrency.
- Log in to comment
@Benjamin Jakimow and I tried to get multiprocessing running in QGIS, but we failed. I will remove the number of processes widget.