The ssh-call occurs within the download.sh script #SBATCH -t 2-00:00:00 # 2 days, example job duration #SBATCH -mem 4GB # memory proportional to 1 cpu #SBATCH -n 1 # 1 task is enough for the ssh The resource requirement is modest for this task, and requires only 1 cpu and will not require significant memory. The slurm job submission script specifies the resource needed to run a download script that will make the ssh-call to the data transfer node. The script structure can be a little bit recursive, and so an outline is included here. The following IPs will access the 'Data Science Network', providing the greatest throughput: The data transfer nodes have multiple interface addresses. The compute node job may perform further tasks on the downloaded data. The examples below stop the gdc-client after two days (172,800 seconds) with the timeout command. The gdc-client process must be limited to a finite execution time to avoid failed transfers overloading the transfer nodes. Gdc-client, running on the data transfer node, will establish up to the maximum number of processes, depending on the number of processes active on the data transfer node. The ssh session call must specify the linux environment to run the gdc-client process, and will include a flag to determine the maximum number of gdc-client processes to run in parallel to accomplish connections to the gdc server at NCI GDC in Chicago. This compute node job is a shell script that includes an ssh session call to a data transfer node (see example call in 'gdc-client availability' section). The job organizing the download (providing access to gdc token, and manifest files) runs on a compute node in the cluster. Jobs are submitted through the SLURM scheduler of the HPC cluster That is to take advantage of the science data firewall rules that have been arranaged to avoid packet-level inspection of large-scale data. The actual download needs to make use of a data transfer node, such as. Understanding the workflow to optimize data transfers

X2go download forbidden