numpy.linalg.linalg.LinAlgError: SVD did not converge

[begumbabur]

What happened?

When running fmriprep on some of my subjects fail to run due to SVD not converging for tcompcor, acompcor, and crowncompcor: numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares.The subjects that fail all fail for one specific task and have no problems in the other tasks. The error I keep receiving for tcompcor is below, I get the exact same error for acompcor and crowncompcor.

Previous posts here have suggested resubmitting and increasing memory. I have tried resubmitting the jobs for the failed subjects over 10 times and some subjects cleared but some consistently are showing the same errors. We're running this on a high performance computing system and it seems to be only using about 35% of the allocated memory, so I'm not sure increasing memory will help in this case.

Has anybody else found alternative ways to fix this issue?

Alternatively, we're thinking if we could skip SVD convergence associated with computing noise components, this can allow us run all subjects without errors. Any suggestions on how to skip this procedure?

What command did you use?

Used singularity to run fmriprep as a batch job with slurm. 
...
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow
...
singularity run --cleanenv \
    --bind ${FMRIPREP_RESOURCES_PATH}:/resources \
    --bind ${BIDS_DIR}:/data \
    --bind ${WORK_DIR}:/work \
    --bind ${OUTPUT_DIR}:/output \
    --bind ${FMRIPREP_RESOURCES_PATH}/.cache/fmriprep:/home/fmriprep/.cache/fmriprep \
    ${FMRIPREP_RESOURCES_PATH}/fmriprep-23.2.0.simg /data /output \
    participant --participant_label $PARTICIPANT_LABEL \
    -w /work \
    -vvv --debug all \
    --notrack \
    --n_cpus 4 --omp-nthreads 1 --mem-mb 22000 \
    --write-graph \
    --ignore slicetiming \
    --output-spaces MNI152NLin6Asym MNI152NLin2009cAsym \
    --fs-license-file /resources/license.txt \
    --skip-bids-validation \
    --fs-no-reconall # if skipping surface reconstruction

What version of fMRIPrep are you running?

fmriprep-23.2.0

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

Node Name: fmriprep_23_2_wf.sub_052_wf.bold_ses_1_task_Video_run_2_wf.bold_confounds_wf.tcompcor

File: /output/sub-052/log/20240521-194119_e6b07c14-dfb0-4056-a899-263120a9c792/crash-20240521-201945-babur-tcompcor-4cfbbe61-15d1-4fac-9fdf-3b7d2bd2afb0.txt
Working Directory: /work/fmriprep_23_2_wf/sub_052_wf/bold_ses_1_task_Video_run_2_wf/bold_confounds_wf/tcompcor
Inputs:
components_file: tcompcor.tsv
failure_mode: NaN
header_prefix: t_comp_cor_
high_pass_cutoff: 128.0
ignore_initial_volumes: 1
mask_files: ['/work/fmriprep_23_2_wf/sub_052_wf/bold_ses_1_task_Video_run_2_wf/bold_fit_wf/unwarp_wf/brainextraction_wf/masker/clipped_mask.nii.gz']
mask_index:
mask_names:
merge_method:
num_components:
percentile_threshold: 0.02
pre_filter: cosine
realigned_file: /work/fmriprep_23_2_wf/sub_052_wf/bold_ses_1_task_Video_run_2_wf/bold_native_wf/boldref_bold/sub-052_ses-1_task-Video_run-2_boldresampled.nii.gz
regress_poly_degree: 1
repetition_time: 1000.0
save_metadata: True
save_pre_filter: True
use_regress_poly:
variance_threshold: 0.5
Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/linear.py", line 47, in run
    node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node tcompcor.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/patches.py", line 66, in _run_interface
	    runtime = super()._run_interface(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/algorithms/confounds.py", line 669, in _run_interface
	    components, filter_basis, metadata = compute_noise_components(
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/algorithms/confounds.py", line 1423, in compute_noise_components
	    voxel_timecourses, basis = cosine_filter(
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/algorithms/confounds.py", line 1203, in cosine_filter
	    betas = np.linalg.lstsq(X, data.T)[0]
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 2326, in lstsq
	    x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 124, in _raise_linalgerror_lstsq
	    raise LinAlgError("SVD did not converge in Linear Least Squares")
	numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares

Additional information / screenshots

No response