Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sdata object has no cellpose or baysor boundaries #64

Open
altunbulakli opened this issue May 13, 2024 · 15 comments
Open

sdata object has no cellpose or baysor boundaries #64

altunbulakli opened this issue May 13, 2024 · 15 comments

Comments

@altunbulakli
Copy link

Hello to the SOPA creators.

I am quite a beginner to Python and the Command Line Interface. I have attempted to run the Ready made Snakemake pipeline for Xenium data (with the Baysor cell segmentation feature)

However, i get the following warning and the error in the resolve_baysor section of the pipeline:

[WARNING] (sopa._sdata) sdata object has no cellpose boundaries and no baysor boundaries. Consider running segmentation first.

and the error:

Finished job 3.
3 of 8 steps (38%) done
MissingInputException in rule resolve_baysor in file C:\Windows\System32\sopa\workflow\Snakefile, line 125:
Missing input files for rule resolve_baysor:
output: C:/xenium_run/TC_070.zarr/.sopa_cache/baysor_boundaries_done, C:/xenium_run/TC_070.zarr/.sopa_cache/table
affected files:

I was thinking that the Snakemake pipeline would run Baysor segmentation automatically.
I also get a similar error when i try a Cellpose only config file or a Baysor.only config file.
What could be the solution here? I have installed both the cellpose package (as part of the sopa installation) and the Baysor package according to their instructions.

Thank you very much for your help :)

@quentinblampey
Copy link
Collaborator

Hello @altunbulakli,

Indeed Sopa should run baysor, but in your case it seems it didn't. Can you show me the full log?

You should have a text file called TC_070.zarr/.sopa_cache/patches_file_baysor, can you show me how it looks? It should contain a list of patch indices. Maybe it is empty for some reasons

@altunbulakli
Copy link
Author

Hey Quentin,

Thank you very much for your help!
The patches_file_baysor looks like this, Indeed it is empty.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
21
22
23
24

@quentinblampey
Copy link
Collaborator

Actually, it looks as expected (it just a list of patches IDs). Each ID corresponds to the name of a directory inside TC_070.zarr/.sopa_cache/baysor_boundaries, do you have such directories?

Also, have you tried the toy example? Do you have the same issue?

@altunbulakli
Copy link
Author

Sorry for the late reply. I wanted to make sure that Baysor was installed correctly and Cellpose was working outside of the Snakemake pipeline:

However, both the Baysor only config, Cellpose only config and the Toy Example command give a similar error:
Here you can find the Toy example log:

(sopa) C:\Users\altun\Documents\sopa\workflow>snakemake --config sdata_path=tuto.zarr --configfile=config/toy/uniform_cellpose.yaml --cores 1 --use-conda
Building DAG of jobs...
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job count


aggregate 1
all 1
annotate 1
explorer 1
image_write 1
patchify_cellpose 1
report 1
resolve_cellpose 1
to_spatialdata 1
total 9

Select jobs to execute...

[Thu May 30 11:52:45 2024]
rule to_spatialdata:
output: tuto.zarr/.zgroup
jobid: 5
reason: Missing output files: tuto.zarr/.zgroup
resources: tmpdir=C:\Users\altun\AppData\Local\Temp, mem_mb=128000, mem_mib=122071

Activating conda environment: sopa
[INFO] (sopa.utils.data) Image of size ((4, 2048, 2048)) with 400 cells and 100 transcripts per cell
[INFO] (sopa.io.standardize) Writing the following spatialdata object to tuto.zarr:
SpatialData object with:
├── Images
│ └── 'image': SpatialImage[cyx] (4, 2048, 2048)
├── Points
│ └── 'transcripts': DataFrame with shape: (, 4) (3D points)
└── Shapes
└── 'cells': GeoDataFrame shape: (400, 1) (2D shapes)
with coordinate systems:
▸ 'global', with elements:
image (Images), transcripts (Points), cells (Shapes)
▸ 'microns', with elements:
transcripts (Points)
[Thu May 30 11:52:55 2024]
Finished job 5.
1 of 9 steps (11%) done
Select jobs to execute...

[Thu May 30 11:52:55 2024]
rule image_write:
input: tuto.zarr/.zgroup
output: tuto.explorer/morphology.ome.tif
jobid: 7
reason: Missing output files: tuto.explorer/morphology.ome.tif; Input files updated by another job: tuto.zarr/.zgroup
resources: tmpdir=C:\Users\altun\AppData\Local\Temp, mem_mb=64000, mem_mib=61036, partition=longq

Activating conda environment: sopa
[WARNING] (sopa._sdata) sdata object has no cellpose boundaries and no baysor boundaries. Consider running segmentation first.
[INFO] (sopa.io.explorer.images) Writing multiscale image with procedure=semi-lazy (load in memory when possible)
[INFO] (sopa.io.explorer.images) (Loading image of shape (4, 2048, 2048)) in memory
[INFO] (sopa.io.explorer.images) > Image of shape (4, 2048, 2048)
[INFO] (sopa.io.explorer.images) > Image of shape (4, 1024, 1024)
[INFO] (sopa.io.explorer.images) > Image of shape (4, 512, 512)
[INFO] (sopa.io.explorer.images) > Image of shape (4, 256, 256)
[INFO] (sopa.io.explorer.images) > Image of shape (4, 128, 128)
[INFO] (sopa.io.explorer.images) > Image of shape (4, 64, 64)
[INFO] (sopa.io.explorer.converter) Saved files in the following directory: tuto.explorer
[INFO] (sopa.io.explorer.converter) You can open the experiment with 'open tuto.explorer\experiment.xenium'
[Thu May 30 11:53:01 2024]
Finished job 7.
2 of 9 steps (22%) done
Select jobs to execute...

[Thu May 30 11:53:01 2024]
checkpoint patchify_cellpose:
input: tuto.zarr/.zgroup
output: tuto.zarr/.sopa_cache/patches_file_image, tuto.zarr/.sopa_cache/patches
jobid: 4
reason: Missing output files: tuto.zarr/.sopa_cache/patches_file_image; Input files updated by another job: tuto.zarr/.zgroup
resources: tmpdir=C:\Users\altun\AppData\Local\Temp
DAG of jobs will be updated after completion.

Activating conda environment: sopa
[INFO] (sopa.patches.patches) 4 patches were saved in sdata['sopa_patches']
Touching output file tuto.zarr/.sopa_cache/patches.
[Thu May 30 11:53:07 2024]
Finished job 4.
3 of 9 steps (33%) done
MissingInputException in rule resolve_cellpose in file C:\Users\altun\Documents\sopa\workflow\Snakefile, line 113:
Missing input files for rule resolve_cellpose:
output: tuto.zarr/.sopa_cache/cellpose_boundaries_done
affected files:
tuto.zarr.sopa_cache\cellpose_boundaries\3.parquet
tuto.zarr.sopa_cache\cellpose_boundaries\1.parquet
tuto.zarr.sopa_cache\cellpose_boundaries\2.parquet
tuto.zarr.sopa_cache\cellpose_boundaries\0.parquet

@altunbulakli
Copy link
Author

And here are my logs for a Baysor only config file snakemake run:

(sopa) C:\Users\altun\Documents\sopa\workflow>snakemake --config data_path=C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070 --configfile=config/xenium/baysor_new.yaml --cores 1 --use-conda
SpatialData object path set to default: C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr
To change this behavior, provide --config sdata_path=... when running the snakemake pipeline
Building DAG of jobs...
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job count


aggregate 1
all 1
explorer 1
image_write 1
patchify_baysor 1
report 1
resolve_baysor 1
to_spatialdata 1
total 8

Select jobs to execute...

[Thu May 30 11:57:56 2024]
rule to_spatialdata:
input: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070
output: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup
jobid: 4
reason: Missing output files: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup
resources: tmpdir=C:\Users\altun\AppData\Local\Temp, mem_mb=128000, mem_mib=122071

Activating conda environment: sopa
[INFO] (sopa.io.standardize) Writing the following spatialdata object to C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr:
SpatialData object with:
├── Images
│ └── 'morphology_focus': MultiscaleSpatialImage[cyx] (1, 20598, 22888), (1, 10299, 11444), (1, 5149, 5722), (1, 2574, 2861), (1, 1287, 1430)
└── Points
└── 'transcripts': DataFrame with shape: (, 10) (3D points)
with coordinate systems:
▸ 'global', with elements:
morphology_focus (Images), transcripts (Points)
[Thu May 30 11:58:49 2024]
Finished job 4.
1 of 8 steps (12%) done
Select jobs to execute...

[Thu May 30 11:58:49 2024]
rule image_write:
input: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup
output: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.explorer/morphology.ome.tif
jobid: 6
reason: Missing output files: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.explorer/morphology.ome.tif; Input files updated by another job: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup
resources: tmpdir=C:\Users\altun\AppData\Local\Temp, mem_mb=64000, mem_mib=61036, partition=longq

Activating conda environment: sopa
[WARNING] (sopa._sdata) sdata object has no cellpose boundaries and no baysor boundaries. Consider running segmentation first.
[INFO] (sopa.io.explorer.images) Writing multiscale image with procedure=semi-lazy (load in memory when possible)
[INFO] (sopa.io.explorer.images) (Loading image of shape (1, 20598, 22888)) in memory
[INFO] (sopa.io.explorer.images) > Image of shape (1, 20598, 22888)
[INFO] (sopa.io.explorer.images) > Image of shape (1, 10299, 11444)
[INFO] (sopa.io.explorer.images) > Image of shape (1, 5149, 5722)
[INFO] (sopa.io.explorer.images) > Image of shape (1, 2574, 2861)
[INFO] (sopa.io.explorer.images) > Image of shape (1, 1287, 1430)
[INFO] (sopa.io.explorer.images) > Image of shape (1, 643, 715)
[INFO] (sopa.io.explorer.converter) Saved files in the following directory: C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.explorer
[INFO] (sopa.io.explorer.converter) You can open the experiment with 'open C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.explorer\experiment.xenium'
[Thu May 30 11:59:12 2024]
Finished job 6.
2 of 8 steps (25%) done
Select jobs to execute...

[Thu May 30 11:59:12 2024]
checkpoint patchify_baysor:
input: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup
output: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.sopa_cache/patches_file_baysor, C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.sopa_cache/baysor_boundaries
jobid: 3
reason: Missing output files: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.sopa_cache/patches_file_baysor; Input files updated by another job: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.zgroup
resources: tmpdir=C:\Users\altun\AppData\Local\Temp
DAG of jobs will be updated after completion.

Activating conda environment: sopa
[INFO] (sopa.patches.patches) Writing sub-CSV for baysor
[########################################] | 100% Completed | 56.31 s
[INFO] (sopa.patches.patches) Patches saved in directory C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries
[INFO] (sopa.patches.patches) Patch 0 has < 4000 transcripts. Baysor will not be run on it.
[INFO] (sopa.patches.patches) Patch 20 has < 4000 transcripts. Baysor will not be run on it.
[Thu May 30 12:00:16 2024]
Finished job 3.
3 of 8 steps (38%) done
MissingInputException in rule resolve_baysor in file C:\Users\altun\Documents\sopa\workflow\Snakefile, line 125:
Missing input files for rule resolve_baysor:
output: C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.sopa_cache/baysor_boundaries_done, C:/Users/altun/Desktop/Xenium_analysis_May_2024/xenium_run/TC_070.zarr/.sopa_cache/table
affected files:
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\21\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\17\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\4\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\24\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\16\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\18\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\11\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\23\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\1\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\13\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\18\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\16\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\24\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\14\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\13\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\11\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\19\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\10\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\7\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\15\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\15\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\1\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\8\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\17\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\5\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\8\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\23\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\22\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\12\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\3\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\4\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\19\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\2\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\12\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\9\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\9\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\6\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\3\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\6\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\5\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\10\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\21\segmentation_counts.loom
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\22\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\2\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\7\segmentation_polygons.json
C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries\14\segmentation_polygons.json

@altunbulakli
Copy link
Author

At the end of this run, we have a C:\Users\altun\Desktop\Xenium_analysis_May_2024\xenium_run\TC_070.zarr.sopa_cache\baysor_boundaries
folder with the 0 to 24 patches.

Here is one of the folder's contents
transcripts.csv

@quentinblampey
Copy link
Collaborator

Weirdly enough, it seems that snakemake uses an incorrect path. Indeed, in your logs I see tuto.zarr.sopa_cache\cellpose_boundaries\3.parquet, but there is a missing \, since it should be tuto.zarr\.sopa_cache\cellpose_boundaries\3.parquet

I don't understand why this happens, and all the other paths look correct, for instance tuto.zarr/.sopa_cache/patches

Just to make sure this is related to Snakemake, can you try to use the CLI as in this tutorial?

@altunbulakli
Copy link
Author

Yes! I was able to follow the CLI tutorial, and it seems to segment the cells, resolve them and aggregate them successfully.
Apart from a small error in report.html writing. I can also visualize the tutorial in Xenium explorer.

So it seems like the Snakemake pipeline that i have (I didnt modify the Snakefile so far) seems to have an error.
One step closer :)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa read . --sdata-path tuto.zarr --technology uniform
[INFO] (sopa.utils.data) Image of size ((4, 2048, 2048)) with 400 cells and 100 transcripts per cell
[INFO] (sopa.io.standardize) Writing the following spatialdata object to tuto.zarr:
SpatialData object with:
├── Images
│ └── 'image': SpatialImage[cyx] (4, 2048, 2048)
├── Points
│ └── 'transcripts': DataFrame with shape: (, 4) (3D points)
└── Shapes
└── 'cells': GeoDataFrame shape: (400, 1) (2D shapes)
with coordinate systems:
▸ 'global', with elements:
image (Images), transcripts (Points), cells (Shapes)
▸ 'microns', with elements:
transcripts (Points)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa patchify image tuto.zarr --patch-width-pixel 1500 --patch-overlap-pixel 50
[INFO] (sopa.patches.patches) 4 patches were saved in sdata['sopa_patches']

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa segmentation cellpose tuto.zarr --channels DAPI --diameter 35 --min-area 2000 --patch-index 0
[INFO] (sopa.segmentation.shapes) Percentage of non-geometrized cells: 0.93% (usually due to segmentation artefacts)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa segmentation cellpose tuto.zarr --channels DAPI --diameter 35 --min-area 2000 --patch-index 1
[INFO] (sopa.segmentation.shapes) Percentage of non-geometrized cells: 0.00% (usually due to segmentation artefacts)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa segmentation cellpose tuto.zarr --channels DAPI --diameter 35 --min-area 2000 --patch-index 2
[INFO] (sopa.segmentation.shapes) Percentage of non-geometrized cells: 2.30% (usually due to segmentation artefacts)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa segmentation cellpose tuto.zarr --channels DAPI --diameter 35 --min-area 2000 --patch-index 3
[INFO] (sopa.segmentation.shapes) Percentage of non-geometrized cells: 0.00% (usually due to segmentation artefacts)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa resolve cellpose tuto.zarr
Reading patches: 100%|███████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 82.79it/s]
[INFO] (sopa.segmentation.stainings) Found 390 total cells
Resolving conflicts: 100%|███████████████████████████████████████████████████████████| 92/92 [00:00<00:00, 4842.21it/s]
[INFO] (sopa.segmentation.stainings) Added 370 cell boundaries in sdata['cellpose_boundaries']

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa aggregate tuto.zarr --gene-column genes --average-intensities --min-transcripts 10
[INFO] (sopa.segmentation.aggregate) Aggregating transcripts over 370 cells
[########################################] | 100% Completed | 103.88 ms
[INFO] (sopa.segmentation.aggregate) Filtering 0 cells
C:\ProgramData\miniconda3\envs\sopa\lib\site-packages\spatialdata_core_elements.py:92: UserWarning: Key cellpose_boundaries already exists. Overwriting it.
self._check_key(key, self.keys(), self._shared_keys)
[INFO] (sopa.segmentation.aggregate) Averaging channels intensity over 370 cells with expansion 0.0
[########################################] | 100% Completed | 211.07 ms
C:\Users\altun\Documents\sopa\sopa\segmentation\aggregate.py:268: ImplicitModificationWarning: Setting element .obsm['intensities'] of view, initializing view as actual.
self.table.obsm[SopaKeys.INTENSITIES_OBSM] = pd.DataFrame(
C:\ProgramData\miniconda3\envs\sopa\lib\site-packages\spatialdata_core_elements.py:92: UserWarning: Key cellpose_boundaries already exists. Overwriting it.
self._check_key(key, self.keys(), self._shared_keys)

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa report tuto.zarr report.html
[INFO] (sopa.io.report.generate) Writing general_section
[INFO] (sopa.io.report.generate) Writing cell_section
[INFO] (sopa.io.report.generate) Writing channel_section
[INFO] (sopa.io.report.generate) Writing transcripts_section
[INFO] (sopa.io.report.generate) Writing representation_section
[INFO] (sopa.io.report.generate) Computing UMAP on 370 cells
[INFO] (sopa.io.report.generate) Writing report to report.html
┌─────────────────────────────── Traceback (most recent call last) ────────────────────────────────┐
│ C:\Users\altun\Documents\sopa\sopa\cli\app.py:212 in report │
│ │
│ 209 │ │
│ 210 │ sdata = read_zarr_standardized(sdata_path) │
│ 211 │ │
│ > 212 │ write_report(path, sdata) │
│ 213 │
│ │
│ ┌─────────────────────────────────────────── locals ───────────────────────────────────────────┐ │
│ │ path = 'report.html' │ │
│ │ read_zarr_standardized = <function read_zarr_standardized at 0x000001BC80AC77F0> │ │
│ │ sdata = SpatialData object with: │ │
│ │ ├── Images │ │
│ │ │ └── 'image': SpatialImage[cyx] (4, 2048, 2048) │ │
│ │ ├── Points │ │
│ │ │ └── 'transcripts': DataFrame with shape: (40000, 4) (3D │ │
│ │ points) │ │
│ │ ├── Shapes │ │
│ │ │ ├── 'cellpose_boundaries': GeoDataFrame shape: (370, 1) (2D │ │
│ │ shapes) │ │
│ │ │ ├── 'cells': GeoDataFrame shape: (400, 1) (2D shapes) │ │
│ │ │ └── 'sopa_patches': GeoDataFrame shape: (4, 1) (2D shapes) │ │
│ │ └── Tables │ │
│ │ │ └── 'table': AnnData (370, 5) │ │
│ │ with coordinate systems: │ │
│ │ ▸ 'global', with elements: │ │
│ │ │ │ image (Images), transcripts (Points), cellpose_boundaries │ │
│ │ (Shapes), cells (Shapes), sopa_patches (Shapes) │ │
│ │ ▸ 'microns', with elements: │ │
│ │ │ │ transcripts (Points) │ │
│ │ sdata_path = 'tuto.zarr' │ │
│ │ write_report = <function write_report at 0x000001BC852F2A70> │ │
│ └──────────────────────────────────────────────────────────────────────────────────────────────┘ │
│ │
│ C:\Users\altun\Documents\sopa\sopa\io\report\generate.py:47 in write_report │
│ │
│ 44 │ sections = SectionBuilder(sdata).compute_sections() │
│ 45 │ │
│ 46 │ log.info(f"Writing report to {path}") │
│ > 47 │ Root(sections).write(path) │
│ 48 │
│ 49 │
│ 50 def _kdeplot_vmax_quantile(values: np.ndarray, quantile: float = 0.95): │
│ │
│ ┌─────────────────────────────────────────── locals ───────────────────────────────────────────┐ │
│ │ path = 'report.html' │ │
│ │ sdata = SpatialData object with: │ │
│ │ ├── Images │ │
│ │ │ └── 'image': SpatialImage[cyx] (4, 2048, 2048) │ │
│ │ ├── Points │ │
│ │ │ └── 'transcripts': DataFrame with shape: (40000, 4) (3D points) │ │
│ │ ├── Shapes │ │
│ │ │ ├── 'cellpose_boundaries': GeoDataFrame shape: (370, 1) (2D shapes) │ │
│ │ │ ├── 'cells': GeoDataFrame shape: (400, 1) (2D shapes) │ │
│ │ │ └── 'sopa_patches': GeoDataFrame shape: (4, 1) (2D shapes) │ │
│ │ └── Tables │ │
│ │ │ └── 'table': AnnData (370, 5) │ │
│ │ with coordinate systems: │ │
│ │ ▸ 'global', with elements: │ │
│ │ │ │ image (Images), transcripts (Points), cellpose_boundaries (Shapes), cells │ │
│ │ (Shapes), sopa_patches (Shapes) │ │
│ │ ▸ 'microns', with elements: │ │
│ │ │ │ transcripts (Points) │ │
│ │ sections = [ │ │
│ │ │ <sopa.io.report.engine.Section object at 0x000001BC857897E0>, │ │
│ │ │ <sopa.io.report.engine.Section object at 0x000001BC879D76A0>, │ │
│ │ │ <sopa.io.report.engine.Section object at 0x000001BC8A53E170>, │ │
│ │ │ <sopa.io.report.engine.Section object at 0x000001BC8A8573A0>, │ │
│ │ │ <sopa.io.report.engine.Section object at 0x000001BC8EE92740> │ │
│ │ ] │ │
│ └──────────────────────────────────────────────────────────────────────────────────────────────┘ │
│ │
│ C:\Users\altun\Documents\sopa\sopa\io\report\engine.py:264 in write │
│ │
│ 261 │ │ self.sanity_check() │
│ 262 │ │ │
│ 263 │ │ with open(path, "w") as f: │
│ > 264 │ │ │ f.write(str(self)) │
│ 265 │ │
│ 266 │ def str(self) -> str: │
│ 267 │ │ return f""" │
│ │
│ ┌───────────────────────────────── locals ─────────────────────────────────┐ │
│ │ f = <_io.TextIOWrapper name='report.html' mode='w' encoding='cp1252'> │ │
│ │ path = 'report.html' │ │
│ │ self = <sopa.io.report.engine.Root object at 0x000001BC85623400> │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ │
│ C:\ProgramData\miniconda3\envs\sopa\lib\encodings\cp1252.py:19 in encode │
│ │
│ 16 │
│ 17 class IncrementalEncoder(codecs.IncrementalEncoder): │
│ 18 │ def encode(self, input, final=False): │
│ > 19 │ │ return codecs.charmap_encode(input,self.errors,encoding_table)[0] │
│ 20 │
│ 21 class IncrementalDecoder(codecs.IncrementalDecoder): │
│ 22 │ def decode(self, input, final=False): │
│ │
│ ┌─────────────────────────────────────────── locals ───────────────────────────────────────────┐ │
│ │ final = False │ │
│ │ input = '\r\n \r\n \r\n \r\n <meta │ │
│ │ charset="utf-8" />\r'+342740 │ │
│ │ self = <encodings.cp1252.IncrementalEncoder object at 0x000001BC8EEAFE80> │ │
│ └──────────────────────────────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘
UnicodeEncodeError: 'charmap' codec can't encode characters in position 209553-209555: character maps to

(sopa) C:\Users\altun\Documents\sopa\workflow>sopa explorer write tuto.zarr --gene-column genes
[INFO] (sopa.io.explorer.table) Writing table with 5 columns
[INFO] (sopa.io.explorer.table) Writing 2 cell categories: region, slide
[INFO] (sopa.io.explorer.shapes) Writing 370 cell polygons
[INFO] (sopa.io.explorer.points) Writing 40000 transcripts
[INFO] (sopa.io.explorer.points) > Level 0: 40000 transcripts
[INFO] (sopa.io.explorer.points) > Level 1: 10000 transcripts
[INFO] (sopa.io.explorer.images) Writing multiscale image with procedure=semi-lazy (load in memory when possible)
[INFO] (sopa.io.explorer.images) (Loading image of shape (4, 2048, 2048)) in memory
[INFO] (sopa.io.explorer.images) > Image of shape (4, 2048, 2048)
[INFO] (sopa.io.explorer.images) > Image of shape (4, 1024, 1024)
[INFO] (sopa.io.explorer.images) > Image of shape (4, 512, 512)
[INFO] (sopa.io.explorer.images) > Image of shape (4, 256, 256)
[INFO] (sopa.io.explorer.images) > Image of shape (4, 128, 128)
[INFO] (sopa.io.explorer.images) > Image of shape (4, 64, 64)
[INFO] (sopa.io.explorer.converter) Saved files in the following directory: tuto.explorer
[INFO] (sopa.io.explorer.converter) You can open the experiment with 'open tuto.explorer\experiment.xenium'

@quentinblampey
Copy link
Collaborator

Good to hear that the CLI works.
I changed something in the way to handle the paths in the snakemake pipeline. Maybe it will fix this "missing /" issue, but since I can't reproduce the issue, I'm not sure. This will be available in the next version of sopa!

Same for the report, I just changed the file encoding, should be fixed in the next release of sopa. I will let you know when it's released!

@quentinblampey
Copy link
Collaborator

I just released sopa==1.1.0, it should fix at least the encoding issue, can you check?

@josenimo
Copy link

Dear @quentinblampey and @altunbulakli,

I am receiving a similar error while trying to process a subset dataset.

Details:
I am trying to run sopa through snakemake locally on a virtual windows machine.
I have two mamba environments, one for snakemake, and one with sopa.
From the snakemake env I run snakemake --config data_path="C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.tif" --configfile="C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\ometif.yaml" --cores 10 --use-conda

It loads the image, but at checkpoint patchify_cellpose it fails claiming that MissingInputException in rule resolve_cellpose in file C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\Snakefile, line 154: Missing input files for rule resolve_cellpose:

The snakefile remains unchanged from sopa repo.

Happy to provide more details if needed :)

config.yaml
read:
  technology: aicsimageio

patchify:
  patch_width_pixel: 2000
  patch_overlap_pixel: 100

segmentation:
  cellpose:
    diameter: 25
    channels: ["DAPI_bg"]
    flow_threshold: 1
    cellprob_threshold: -6
    min_area: 300
    model_type: "nuclei"
    clip_limit: 0.2
    gaussian_sigma: 1

aggregate:
  average_intensities: true
  min_intensity_ratio: 0.1
  expand_radius_ratio: 0.1
StackTrace
(snakemake) C:\Users\jnimoca\Desktop\SOPA\sopa>snakemake --config data_path="C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.tif" --configfile="C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\ometif.yaml" --cores 10 --use-conda
SpatialData object path set to default: C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr
To change this behavior, provide `--config sdata_path=...` when running the snakemake pipeline
Building DAG of jobs...
Provided cores: 10
Rules claiming more threads will be scaled down.
Job stats:
job                  count
-----------------  -------
aggregate                1
all                      1
explorer                 1
image_write              1
patchify_cellpose        1
report                   1
resolve_cellpose         1
to_spatialdata           1
total                    8

Select jobs to execute...

[Thu Jul 25 15:05:35 2024]
rule to_spatialdata:
    input: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.tif
    output: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup
    jobid: 4
    reason: Missing output files: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup
    resources: tmpdir=C:\Users\jnimoca\AppData\Local\Temp\4, mem_mb=128000, mem_mib=122071

Activating conda environment: sopa
[WARNING] (sopa.io.standardize) No transcripts found. Some tools from sopa will not be available.
[INFO] (sopa.io.standardize) Writing the following spatialdata object to C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr:
SpatialData object
└── Images
      └── 'image': DataTree[cyx] (15, 8000, 8000), (15, 4000, 4000), (15, 2000, 2000), (15, 1000, 1000), (15, 500, 500)
with coordinate systems:
    ▸ 'global', with elements:
        image (Images)
INFO     The Zarr backing store has been changed from None the new file path: C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr
[Thu Jul 25 15:07:44 2024]
Finished job 4.
1 of 8 steps (12%) done
Select jobs to execute...

[Thu Jul 25 15:07:44 2024]
rule image_write:
    input: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup
    output: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.explorer/morphology.ome.tif
    jobid: 6
    reason: Missing output files: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.explorer/morphology.ome.tif; Input files updated by another job: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup
    resources: tmpdir=C:\Users\jnimoca\AppData\Local\Temp\4, mem_mb=64000, mem_mib=61036, partition=longq

Activating conda environment: sopa

[Thu Jul 25 15:07:44 2024]
checkpoint patchify_cellpose:
    input: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup
    output: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.sopa_cache/patches_file_image, C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.sopa_cache/patches
    jobid: 3
    reason: Missing output files: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.sopa_cache/patches_file_image; Input files updated by another job: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.zgroup
    resources: tmpdir=C:\Users\jnimoca\AppData\Local\Temp\4
DAG of jobs will be updated after completion.

Activating conda environment: sopa
[WARNING] (sopa._sdata) sdata object has no valid segmentation boundary. Consider running Sopa segmentation first.
[INFO] (sopa.patches.patches) 25 patches were saved in sdata['sopa_patches']
[INFO] (sopa.io.explorer.images) Writing multiscale image with procedure=semi-lazy (load in memory when possible)
[INFO] (sopa.io.explorer.images)    (Loading image of shape (15, 8000, 8000)) in memory
Touching output file C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.sopa_cache/patches.
[Thu Jul 25 15:07:56 2024]
Finished job 3.
2 of 8 steps (25%) done
MissingInputException in rule resolve_cellpose in file C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\Snakefile, line 154:
Missing input files for rule resolve_cellpose:
    output: C:/Users/jnimoca/Desktop/SOPA/sopa/workflow/data/991_subset.ome.zarr/.sopa_cache/cellpose_boundaries_done
    affected files:
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\10.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\15.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\22.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\20.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\6.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\19.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\8.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\16.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\13.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\24.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\0.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\14.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\3.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\21.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\18.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\1.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\2.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\11.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\9.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\4.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\5.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\23.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\12.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\7.parquet
        C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\17.parquet
[INFO] (sopa.io.explorer.images)    > Image of shape (15, 8000, 8000)
[INFO] (sopa.io.explorer.images)    > Image of shape (15, 4000, 4000)
[INFO] (sopa.io.explorer.images)    > Image of shape (15, 2000, 2000)
[INFO] (sopa.io.explorer.images)    > Image of shape (15, 1000, 1000)
[INFO] (sopa.io.explorer.images)    > Image of shape (15, 500, 500)
[INFO] (sopa.io.explorer.images)    > Image of shape (15, 250, 250)
[INFO] (sopa.io.explorer.converter) Saved files in the following directory: C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.explorer
[INFO] (sopa.io.explorer.converter) You can open the experiment with 'open C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.explorer\experiment.xenium'
sopa mamba list
(sopa) C:\Users\jnimoca\Desktop\SOPA\sopa>mamba list
# packages in environment at C:\Users\jnimoca\AppData\Local\miniforge3\envs\sopa:
#
# Name                    Version                   Build  Channel
aicsimageio               4.14.0                   pypi_0    pypi
aiobotocore               2.11.2                   pypi_0    pypi
aiohttp                   3.9.5                    pypi_0    pypi
aioitertools              0.11.0                   pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
anndata                   0.10.8                   pypi_0    pypi
annotated-types           0.7.0                    pypi_0    pypi
array-api-compat          1.7.1                    pypi_0    pypi
asciitree                 0.3.3                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
botocore                  1.34.19                  pypi_0    pypi
bzip2                     1.0.8                h2466b09_7    conda-forge
ca-certificates           2024.7.4             h56e8100_0    conda-forge
cellpose                  3.0.10                   pypi_0    pypi
certifi                   2024.7.4                 pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
colorcet                  3.1.0                    pypi_0    pypi
contourpy                 1.2.1                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
dask                      2024.7.1                 pypi_0    pypi
dask-expr                 1.1.9                    pypi_0    pypi
dask-image                2024.5.3                 pypi_0    pypi
datashader                0.16.3                   pypi_0    pypi
distributed               2024.7.1                 pypi_0    pypi
elementpath               4.4.0                    pypi_0    pypi
exceptiongroup            1.2.2                    pypi_0    pypi
fasteners                 0.19                     pypi_0    pypi
fastremap                 1.15.0                   pypi_0    pypi
fcsparser                 0.2.8                    pypi_0    pypi
filelock                  3.15.4                   pypi_0    pypi
flake8                    7.1.0                    pypi_0    pypi
fonttools                 4.53.1                   pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2023.6.0                 pypi_0    pypi
geopandas                 1.0.1                    pypi_0    pypi
h5py                      3.11.0                   pypi_0    pypi
idna                      3.7                      pypi_0    pypi
imagecodecs               2024.6.1                 pypi_0    pypi
imageio                   2.34.2                   pypi_0    pypi
importlib-metadata        8.0.0                    pypi_0    pypi
intel-openmp              2021.4.0                 pypi_0    pypi
jinja2                    3.1.4                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
joblib                    1.4.2                    pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
lamin-utils               0.13.2                   pypi_0    pypi
lazy-loader               0.4                      pypi_0    pypi
legacy-api-wrap           1.4                      pypi_0    pypi
libffi                    3.4.2                h8ffe710_5    conda-forge
libsqlite                 3.46.0               h2466b09_0    conda-forge
libzlib                   1.3.1                h2466b09_1    conda-forge
llvmlite                  0.43.0                   pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
lxml                      4.9.4                    pypi_0    pypi
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.9.1                    pypi_0    pypi
mccabe                    0.7.0                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mkl                       2021.4.0                 pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
msgpack                   1.0.8                    pypi_0    pypi
multidict                 6.0.5                    pypi_0    pypi
multipledispatch          1.0.0                    pypi_0    pypi
multiscale-spatial-image  1.0.0                    pypi_0    pypi
natsort                   8.4.0                    pypi_0    pypi
networkx                  3.3                      pypi_0    pypi
numba                     0.60.0                   pypi_0    pypi
numcodecs                 0.13.0                   pypi_0    pypi
numpy                     1.26.4                   pypi_0    pypi
ome-types                 0.5.1.post1              pypi_0    pypi
ome-zarr                  0.9.0                    pypi_0    pypi
opencv-python             4.10.0.84                pypi_0    pypi
opencv-python-headless    4.10.0.84                pypi_0    pypi
openssl                   3.3.1                h2466b09_2    conda-forge
packaging                 24.1                     pypi_0    pypi
pandas                    2.2.2                    pypi_0    pypi
param                     2.1.1                    pypi_0    pypi
partd                     1.4.2                    pypi_0    pypi
patsy                     0.5.6                    pypi_0    pypi
pillow                    10.4.0                   pypi_0    pypi
pims                      0.7                      pypi_0    pypi
pip                       24.0               pyhd8ed1ab_0    conda-forge
platformdirs              4.2.2                    pypi_0    pypi
pooch                     1.8.2                    pypi_0    pypi
psutil                    6.0.0                    pypi_0    pypi
pyarrow                   17.0.0                   pypi_0    pypi
pycodestyle               2.12.0                   pypi_0    pypi
pyct                      0.5.0                    pypi_0    pypi
pydantic                  2.8.2                    pypi_0    pypi
pydantic-compat           0.1.2                    pypi_0    pypi
pydantic-core             2.20.1                   pypi_0    pypi
pyflakes                  3.2.0                    pypi_0    pypi
pygments                  2.18.0                   pypi_0    pypi
pynndescent               0.5.13                   pypi_0    pypi
pyogrio                   0.9.0                    pypi_0    pypi
pyparsing                 3.1.2                    pypi_0    pypi
pyproj                    3.6.1                    pypi_0    pypi
python                    3.10.0          hcf16a7b_3_cpython    conda-forge
python-dateutil           2.9.0.post0              pypi_0    pypi
pytz                      2024.1                   pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readfcs                   1.1.8                    pypi_0    pypi
requests                  2.32.3                   pypi_0    pypi
resource-backed-dask-array 0.1.0                    pypi_0    pypi
rich                      13.7.1                   pypi_0    pypi
roifile                   2024.5.24                pypi_0    pypi
s3fs                      0.6.0                    pypi_0    pypi
scanpy                    1.10.2                   pypi_0    pypi
scikit-image              0.24.0                   pypi_0    pypi
scikit-learn              1.5.1                    pypi_0    pypi
scipy                     1.14.0                   pypi_0    pypi
seaborn                   0.13.2                   pypi_0    pypi
session-info              1.0.0                    pypi_0    pypi
setuptools                71.0.4             pyhd8ed1ab_0    conda-forge
shapely                   2.0.5                    pypi_0    pypi
shellingham               1.5.4                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
slicerator                1.1.0                    pypi_0    pypi
sopa                      1.1.1                    pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
spatial-image             1.1.0                    pypi_0    pypi
spatialdata               0.2.1                    pypi_0    pypi
spatialdata-io            0.1.3.post0              pypi_0    pypi
sqlite                    3.46.0               h2466b09_0    conda-forge
statsmodels               0.14.2                   pypi_0    pypi
stdlib-list               0.10.0                   pypi_0    pypi
sympy                     1.13.1                   pypi_0    pypi
tbb                       2021.13.0                pypi_0    pypi
tblib                     3.0.0                    pypi_0    pypi
threadpoolctl             3.5.0                    pypi_0    pypi
tifffile                  2023.2.28                pypi_0    pypi
tk                        8.6.13               h5226925_1    conda-forge
toolz                     0.12.1                   pypi_0    pypi
torch                     2.3.1                    pypi_0    pypi
tornado                   6.4.1                    pypi_0    pypi
tqdm                      4.66.4                   pypi_0    pypi
typer                     0.12.3                   pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
umap-learn                0.5.6                    pypi_0    pypi
urllib3                   2.0.7                    pypi_0    pypi
vc                        14.3                h8a93ad2_20    conda-forge
vc14_runtime              14.40.33810         ha82c5b3_20    conda-forge
vs2015_runtime            14.40.33810         h3bf8584_20    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
wrapt                     1.16.0                   pypi_0    pypi
xarray                    2024.6.0                 pypi_0    pypi
xarray-dataclasses        1.8.0                    pypi_0    pypi
xarray-datatree           0.0.14                   pypi_0    pypi
xarray-schema             0.0.3                    pypi_0    pypi
xarray-spatial            0.4.0                    pypi_0    pypi
xmlschema                 3.3.1                    pypi_0    pypi
xsdata                    24.3.1                   pypi_0    pypi
xz                        5.2.6                h8d14728_0    conda-forge
yarl                      1.9.4                    pypi_0    pypi
zarr                      2.15.0                   pypi_0    pypi
zict                      3.0.0                    pypi_0    pypi
zipp                      3.19.2                   pypi_0    pypi
snakemake mamba list
(snakemake) C:\Users\jnimoca\Desktop\SOPA\sopa>mamba list
# packages in environment at C:\Users\jnimoca\AppData\Local\miniforge3\envs\snakemake:
#
# Name                    Version                   Build  Channel
aioeasywebdav             2.4.0              pyha770c72_0    conda-forge
aiohttp                   3.9.5           py310h8d17308_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
amply                     0.1.6              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
async-timeout             4.0.3              pyhd8ed1ab_0    conda-forge
attmap                    0.13.2             pyhd8ed1ab_0    conda-forge
attrs                     23.2.0             pyh71513ae_0    conda-forge
bcrypt                    4.2.0           py310hc226416_0    conda-forge
boto3                     1.34.148           pyhd8ed1ab_0    conda-forge
botocore                  1.34.148        pyge310_1234567_0    conda-forge
brotli-python             1.1.0           py310h00ffb61_1    conda-forge
bzip2                     1.0.8                h2466b09_7    conda-forge
c-ares                    1.32.3               h2466b09_0    conda-forge
ca-certificates           2024.7.4             h56e8100_0    conda-forge
cachetools                5.4.0              pyhd8ed1ab_0    conda-forge
certifi                   2024.7.4           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py310h8d17308_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
configargparse            1.7                pyhd8ed1ab_0    conda-forge
connection_pool           0.0.3              pyhd3deb0d_0    conda-forge
cryptography              42.0.8          py310h25a5809_0    conda-forge
datrie                    0.8.2           py310h8d17308_7    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
docutils                  0.21.2             pyhd8ed1ab_0    conda-forge
dpath                     2.2.0              pyha770c72_0    conda-forge
dropbox                   12.0.2             pyhd8ed1ab_0    conda-forge
eido                      0.2.2              pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.2              pyhd8ed1ab_0    conda-forge
filechunkio               1.8                        py_2    conda-forge
frozenlist                1.4.1           py310h8d17308_0    conda-forge
ftputil                   5.1.0              pyhd8ed1ab_0    conda-forge
gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.43             pyhd8ed1ab_0    conda-forge
glpk                      5.0                  h8ffe710_0    conda-forge
google-api-core           2.19.1             pyhd8ed1ab_0    conda-forge
google-api-python-client  2.137.0            pyhd8ed1ab_0    conda-forge
google-auth               2.32.0             pyhff2d567_0    conda-forge
google-auth-httplib2      0.2.0              pyhd8ed1ab_0    conda-forge
google-cloud-core         2.4.1              pyhd8ed1ab_0    conda-forge
google-cloud-storage      2.18.0             pyhff2d567_0    conda-forge
google-crc32c             1.1.2           py310ha7be474_5    conda-forge
google-resumable-media    2.7.0              pyhd8ed1ab_0    conda-forge
googleapis-common-protos  1.63.2             pyhd8ed1ab_0    conda-forge
grpcio                    1.62.2          py310hb84602e_0    conda-forge
h2                        4.1.0              pyhd8ed1ab_0    conda-forge
hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
httplib2                  0.22.0             pyhd8ed1ab_0    conda-forge
humanfriendly             10.0            py310h5588dad_5    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
idna                      3.7                pyhd8ed1ab_0    conda-forge
importlib_resources       6.4.0              pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
intel-openmp              2024.2.0           h57928b3_980    conda-forge
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
jsonschema                4.23.0             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.12.1          pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.2           py310h5588dad_0    conda-forge
libabseil                 20240116.2      cxx17_he0c23c2_1    conda-forge
libblas                   3.9.0              23_win64_mkl    conda-forge
libcblas                  3.9.0              23_win64_mkl    conda-forge
libcrc32c                 1.1.2                h0e60522_0    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libgrpc                   1.62.2               h5273850_0    conda-forge
libhwloc                  2.11.1          default_h8125262_1000    conda-forge
libiconv                  1.17                 hcfcfb64_2    conda-forge
liblapack                 3.9.0              23_win64_mkl    conda-forge
libprotobuf               4.25.3               h503648d_0    conda-forge
libre2-11                 2023.09.01           hf8d8778_2    conda-forge
libsqlite                 3.46.0               h2466b09_0    conda-forge
libxml2                   2.12.7               h0f24e4e_4    conda-forge
libzlib                   1.3.1                h2466b09_1    conda-forge
logmuse                   0.2.6              pyh8c360ce_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.5           py310h8d17308_0    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
mkl                       2024.1.0           h66d3029_694    conda-forge
multidict                 6.0.5           py310h8d17308_0    conda-forge
nbformat                  5.10.4             pyhd8ed1ab_0    conda-forge
numpy                     2.0.0           py310h1ec8c79_0    conda-forge
oauth2client              4.1.3                      py_0    conda-forge
openssl                   3.3.1                h2466b09_2    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
pandas                    2.2.2           py310hb4db72f_1    conda-forge
paramiko                  3.4.0              pyhd8ed1ab_0    conda-forge
peppy                     0.40.4             pyhd8ed1ab_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
plac                      1.4.3              pyhd8ed1ab_0    conda-forge
platformdirs              4.2.2              pyhd8ed1ab_0    conda-forge
pluggy                    1.5.0              pyhd8ed1ab_0    conda-forge
ply                       3.11               pyhd8ed1ab_2    conda-forge
prettytable               3.10.2             pyhd8ed1ab_0    conda-forge
proto-plus                1.23.0             pyhd8ed1ab_0    conda-forge
protobuf                  4.25.3          py310hf536535_0    conda-forge
psutil                    6.0.0           py310ha8f682b_0    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
pulp                      2.7.0           py310h5588dad_1    conda-forge
pyasn1                    0.6.0              pyhd8ed1ab_0    conda-forge
pyasn1-modules            0.4.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.22               pyhd8ed1ab_0    conda-forge
pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
pynacl                    1.5.0           py310h635b8f1_3    conda-forge
pyopenssl                 24.2.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
pyreadline3               3.4.1           py310h5588dad_3    conda-forge
pysftp                    0.2.9                      py_1    conda-forge
pysocks                   1.7.1              pyh0701188_6    conda-forge
pytest                    8.3.1              pyhd8ed1ab_0    conda-forge
python                    3.10.0          hcf16a7b_3_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.20.0             pyhd8ed1ab_0    conda-forge
python-irodsclient        2.0.1              pyhd8ed1ab_0    conda-forge
python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
python_abi                3.10                    4_cp310    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pywin32                   306             py310h00ffb61_2    conda-forge
pyyaml                    6.0.1           py310h8d17308_1    conda-forge
re2                       2023.09.01           hd3b24a8_2    conda-forge
referencing               0.35.1             pyhd8ed1ab_0    conda-forge
requests                  2.32.3             pyhd8ed1ab_0    conda-forge
reretry                   0.11.8             pyhd8ed1ab_0    conda-forge
rich                      13.7.1             pyhd8ed1ab_0    conda-forge
rpds-py                   0.19.1          py310hc226416_0    conda-forge
rsa                       4.9                pyhd8ed1ab_0    conda-forge
s3transfer                0.10.2             pyhd8ed1ab_0    conda-forge
setuptools                71.0.4             pyhd8ed1ab_0    conda-forge
setuptools-scm            8.1.0              pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
slacker                   0.14.0                     py_0    conda-forge
smart_open                7.0.4              pyhd8ed1ab_0    conda-forge
smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
snakemake                 7.32.4               hdfd78af_1    bioconda
snakemake-minimal         7.32.4             pyhdfd78af_1    bioconda
sqlite                    3.46.0               h2466b09_0    conda-forge
stone                     3.3.2              pyhd8ed1ab_0    conda-forge
stopit                    1.1.2                      py_0    conda-forge
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
tbb                       2021.12.0            hc790b64_3    conda-forge
throttler                 1.2.2              pyhd8ed1ab_0    conda-forge
tk                        8.6.13               h5226925_1    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toposort                  1.10               pyhd8ed1ab_0    conda-forge
traitlets                 5.14.3             pyhd8ed1ab_0    conda-forge
typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
ubiquerg                  0.8.0              pyhd8ed1ab_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
uritemplate               4.1.1              pyhd8ed1ab_0    conda-forge
urllib3                   2.2.2              pyhd8ed1ab_1    conda-forge
vc                        14.3                h8a93ad2_20    conda-forge
vc14_runtime              14.40.33810         ha82c5b3_20    conda-forge
veracitools               0.1.3                      py_0    conda-forge
vs2015_runtime            14.40.33810         h3bf8584_20    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
win_inet_pton             1.1.0              pyhd8ed1ab_6    conda-forge
wrapt                     1.16.0          py310h8d17308_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
yaml                      0.2.5                h8ffe710_2    conda-forge
yarl                      1.9.4           py310h8d17308_0    conda-forge
yte                       1.5.4              pyha770c72_0    conda-forge
zipp                      3.19.2             pyhd8ed1ab_0    conda-forge
zstandard                 0.23.0          py310he5e10e1_0    conda-forge
zstd                      1.5.6                h0ea2cb4_0    conda-forge

@quentinblampey
Copy link
Collaborator

Hello @josenimo,

Can you check if the "missing" files are really missing?
E.g., does this file exist?

C:\Users\jnimoca\Desktop\SOPA\sopa\workflow\data\991_subset.ome.zarr\.sopa_cache\cellpose_boundaries\10.parquet

If the file exists, then snakemake is not detecting it for some reasons. I suspect it has something to do with how paths are handled in Windows.

Two questions to check if my assumptions are correct:

  • does it also happen with the toy example from the doc?
  • does it also happen on Linux/MacOS?

@josenimo
Copy link

Hey @quentinblampey ,
thank you for your quick responses.

Update:

  1. Files are actually not there, only found patches and patches_file_image inside .sopa_cache
  2. toy example does not run, and fails in a similar way
  3. It runs perfectly on MacOS

Happy to provide more details to help solve Windows issues.

@quentinblampey
Copy link
Collaborator

Ok thanks for the details. I think it's an issue related to snakemake, which doesn't trigger the rules that create the input needed for the resolve_cellpose rule
Can you tell me which version of snakemake and windows do you have?

When I have access to a windows laptop (i.e. not before late august), I'll check this and try to solve this!

Meanwhile, the CLI and the API should work fine on windows :)

@josenimo
Copy link

snakemake version 7.32.4
Windows Server 2022 Standard
All good, thank you for the info. I almost got it running on SLURM :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants