Cluster problem
Hi,
we (+ @hennies ) tried to convert more datasets and ran into:
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
ERROR: [pid 23554] Worker Worker(salt=654140451, workers=1, host=login.cluster.embl.de, username=schorb, pid=23554) failed CopyVolumeSlurm(tmp_folder=/scratch/civd-em/TMP/tmp_Covid19-S5-mock-Cell1-2, max_jobs=64, config_dir=/scratch/civd-em/TMP/tmp_Covid19-S5-mock-Cell1-2/configs, input_path=/g/emcf/common/5792_Sars-Cov-2/Exp_070420/FIB-SEM/alignments/20-04-25_S5_mock_cell1_2_Phoebe/align2_amst_inv_black_bg, input_key=*.tif*, output_path=./data/Covid19-S5-mock-Cell1-2/images/local/sbem-6dpf-1-whole-raw.n5, output_key=setup0/timepoint0/s0, prefix=initial_scale, dtype=None, fit_to_roi=False, effective_scale_factor=[], dependency=DummyTask)
Is it because of the scratch dir that we like to use for TMP? (note the typo...)
This is funny because the init procedures worked with that directory...
EDIT:
OK, does not seem to be due to scratch:
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
ERROR: [pid 36259] Worker Worker(salt=062125892, workers=1, host=login.cluster.embl.de, username=schorb, pid=36259) failed CopyVolumeSlurm(tmp_folder=/scratch/schorb/covid-em/TMP/tmp_Covid19-S5-mock-Cell1-2, max_jobs=64, config_dir=/scratch/schorb/covid-em/TMP/tmp_Covid19-S5-mock-Cell1-2/configs, input_path=/g/emcf/common/5792_Sars-Cov-2/Exp_070420/FIB-SEM/alignments/20-04-25_S5_mock_cell1_2_Phoebe/align2_amst_inv_black_bg, input_key=*.tif*, output_path=./data/Covid19-S5-mock-Cell1-2/images/local/sbem-6dpf-1-whole-raw.n5, output_key=setup0/timepoint0/s0, prefix=initial_scale, dtype=None, fit_to_roi=False, effective_scale_factor=[], dependency=DummyTask)
Traceback (most recent call last):
File "/g/emcf/common/5792_Sars-Cov-2/covid-em/software/miniconda3/envs/mmb-dev/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
new_deps = self._run_get_new_deps()
File "/g/emcf/common/5792_Sars-Cov-2/covid-em/software/miniconda3/envs/mmb-dev/lib/python3.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
task_gen = self.task.run()
File "/g/emcf/common/5792_Sars-Cov-2/covid-em/software/cluster_tools/cluster_tools/cluster_tasks.py", line 95, in run
raise e
File "/g/emcf/common/5792_Sars-Cov-2/covid-em/software/cluster_tools/cluster_tools/cluster_tasks.py", line 81, in run
self.run_impl()
File "/g/emcf/common/5792_Sars-Cov-2/covid-em/software/cluster_tools/cluster_tools/copy_volume/copy_volume.py", line 138, in run_impl
self.submit_jobs(n_jobs, self.prefix)
File "/g/emcf/common/5792_Sars-Cov-2/covid-em/software/cluster_tools/cluster_tools/cluster_tasks.py", line 479, in submit_jobs
outp = check_output(command).decode().rstrip()
File "/g/emcf/common/5792_Sars-Cov-2/covid-em/software/miniconda3/envs/mmb-dev/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/g/emcf/common/5792_Sars-Cov-2/covid-em/software/miniconda3/envs/mmb-dev/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['sbatch', '-o', '/scratch/schorb/covid-em/TMP/tmp_Covid19-S5-mock-Cell1-2/logs/copy_volume_initial_scale_0.log', '-e', '/scratch/schorb/covid-em/TMP/tmp_Covid19-S5-mock-Cell1-2/error_logs/copy_volume_initial_scale_0.err', '-J', 'copy_volume_initial_scale_0', '/scratch/schorb/covid-em/TMP/tmp_Covid19-S5-mock-Cell1-2/slurm_copy_volume_initial_scale.sh', '0']' returned non-zero exit status 1.
Edited by Martin Schorb