Aspera安装

Ensure that your Linux environment includes the following libraries:

  • OpenSSL version 1.0.2g or higher (see below)
  • Mesa EGL (If you get a launch error about libEGL, see below.)
  • glib2 version 2.28 or higher.

    下载Aspera

    1. wget -c https://d3gcli72yxqn2z.cloudfront.net/connect/v4/bin/ibm-aspera-connect-3.9.9.177872-linux-g2.12-64.tar.gz

    解压

    1. tar -zxvf ibm-aspera-connect-version.tar.gz
    2. ./ibm-aspera-connect-version.sh

    查看软件是否安装成功

    ```bash cd # 去根目录 ls -a

添加环境变量

echo ‘export PATH=~/.aspera/connect/bin:$PATH’ echo ‘export PATH=~/.aspera/connect/bin:$PATH’ >> ~/.bashrc

source ~/.bashrc

ascp -h #查看帮助信息，确认可以正常使用

  1. <a name="DFtEl"></a>
  2. # Ascp命令参考
  3. Ascp (Aspera安全复制) 可执行文件是一个命令行fasp传输程序。本主题涵盖了完整的命令使用，包括一般语法指南、支持的环境变量、概要和命令选项。Ascp (Aspera安全复制) 可执行文件是一个命令行fasp传输程序。本主题涵盖了完整的命令使用，包括一般语法指南、支持的环境变量、概要和命令选项。
  4. <a name="FjvMi"></a>
  5. ## 通用语法指南
  6. | Item | Decription |
  7. | --- | --- |
  8. | symbols used in the paths | Use single-quote (' ') and forward-slashes (/) on all platforms. |
  9. | Characters to avoid | / \\ " : ' ? > < & * &#124; |
  11. <a name="i9O7w"></a>
  12. ## Environment Variables
  13. If needed, you can set the following environment variables for use with the ascp command:
  15. | Item | Initiation Command |
  16. | --- | --- |
  17. | Password | ASPERA_SCP_PASS=password |
  18. | Token | ASPERA_SCP_TOKEN=token |
  19. | Cookie | ASPERA_SCP_COOKIE=cookie |
  20. | Content Protection Password | ASPERA_SCP_FILEPASS=password |
  21. | Proxy Server Password | ASPERA_PROXY_PASS=proxy_server_password |
  23. <a name="NbcxP"></a>
  24. ## Ascp Usage
  25. ```bash
  26. ascp options [[user@]srcHost:]source_file1[,source_file2,...] [[user@]destHost:]target_path


重要提示: 如果您没有为传输指定用户名，本地用户名将被身份验证 (默认情况下)。在Windows计算机和域用户的情况下，传输服务器将从用户名中删除域 (例如，身份验证管理员，而不是域 \ 管理员)。因此，如果适用于用户，您需要明确指定一个域。

Ascp Options

Option Description
-h, —help Display usage.
-A, —version Display version and license information; then exit.
-T Disable encryption for maximum throughput.
-d Create target directory if it doesn’t already exist.
-p Preserve file timestamps for source modification time (mtime) and last access time (atime).Important: On Windows, mtime and atime may be affected when the system automatically adjusts for Daylight Savings Time (DST). For details, see the Microsoft KB article, http://support.microsoft.com/kb/129574.
Important: On Isilon IQ OneFS systems, last access time (atime) is disabled by default (see sysctl efs.bam.atime_enabled). You will see atime is set to be the same as mtime when using -p option. Use the command “sysctl efs.bam.atime_enabled=1” to enable the preservation of atime on your Isilon system.
-q Quiet mode (to disable progress display).
-v Verbose mode (prints connection and authentication debug messages in the log file). For information on log files, see Log Files.
-6 Enable IPv6 address support. When using IPv6, the numeric host can be written inside brackets. For example, [2001:0:4137:9e50:201b:63d3:ba92:da] or [fe80::21b:21ff:fe1c:5072%eth1].
-D | -DD | -DDD Specify the debug level, where each D is an additional level of debugging.
-l max_rate Set the target transfer rate in Kbps (default: 10000 Kbps). If the ascp client does not specify a target rate, it will be acquired from aspera.conf (server-side, as the local aspera.conf target rate setting doesn’t apply). If local or server aspera.conf rate caps are specified, the “starting” (default) rates will be not higher than the cap.
-m min_rate Set the minimum transfer rate in Kbps (efault: 0. If the ascp client does not specify a minimum rate, it will be acquired from aspera.conf (server-side, as the local aspera.conf minimum rate setting doesn’t apply). If local or server aspera.conf rate caps are specified, the “starting” (default) rates will be not higher than the cap.
-u user_string Apply a user string, such as variables for pre- and post-processing.
-i private_key_file Use public key authentication and specify the private key file. Typically, the private key file is in the directory $HOME/.ssh/id_[algorithm].
-w{r|f} Test bandwidth from server to client (r) or client to server (f). Currently a beta option.
-K probe_rate Set probing rate (Kbps) when measuring bottleneck bandwidth.
-k{0|1|2|3} Enable resuming partially transferred files at the specified resume level (default: 0). Note that this must be specified for your first transfer; otherwise, it will not work for subsequent transfers. Resume levels:
- 0 - Always retransfer the entire file.
- 1 - Check file attributes and resume if the current and original attributes match.
- 2 - Check file attributes and do a sparse file checksum; resume if the current and original attributes/checksums match.
- 3 - Check file attributes and do a full file checksum; resume if the current and original attributes/checksums match.

Note that when a complete file exists at the destination (no .aspx), the source file size is compared with the destination file size. When a partial file and a valid .aspx file exist at the destination, the source file size is compared with the file size recorded inside the .aspx file. | | -Z dgramsize | Specify the datagram size (MTU) for _fasp. By default, the detected path MTU is used. (Range: 296 - 10000 bytes)Note: As of version 3.3, datagram size can also be enforced by the server using in aspera.conf. If size is set with both -Z (client side) and (server side), the setting is used. If the client-side is pre-3.3, datagram size is determined by the -Z setting, regardless of the server-side setting for . In this case, if there is no -Z setting, datagram size is based on the discovered MTU and the server logs the message “LOG Peer client doesn’t support alternative datagram size”. | | -g readsize | Set the read-block size, a performance-tuning parameter for an Aspera sender (which _only takes effect if the sender is a server). It represents the maximum number of bytes that can be stored within a block as the block is being transferred from the source disk drive to the receiver. The default of 0 will cause the Aspera sender to use its default internal buffer size, which may be different for different operating systems. Note that 500M (524,288,000 bytes) is the maximum block size. | | -G writesize | This is a performance-tuning parameter for an Aspera receiver (which only takes effect if the receiver is a _server). It represents the maximum bytes within a block that an ascp receiver can write to disk. The default of 0 will cause the Aspera receiver to use its default internal buffer size, which may be different for different operating systems. Note that 500M (524,288,000 bytes) is the maximum block size. | | -L locallog_dir | Specify a logging directory in the local host, instead of using the default directory. | | -R remote_log_dir | Specify a logging directory in the remote host, instead of using the default directory. | | -S remote_ascp | Specify the name of the remote ascp binary (if different). | | -e prepost | Specify an alternate pre/post command. Be sure to use the complete path and file name. | | -O fasp_port | Set the UDP port to be used by _fasp for data transfer. (Default: 33001) | | -P ssh-port | Set the TCP port to be used for fasp session initiation. (Default: 33001) | | -C nid:ncount | Use parallel transfer on a multi-node/core system. Specify the node id (nid) and count (ncount) in the format 1:2, 2:2. Assign each participant to an independent UDP port. | | -E pattern | Exclude files or directories with the specified pattern from the transfer. This option can be used multiple times to exclude many patterns. Up to 16 patterns can be used by using -E. Two symbols can be used in the pattern, as shown below.
- * (asterisk) represents zero or more characters in a string, for example .tmp matches .tmp and abcde.tmp.
- ? (question mark) represents a single character, for example t?p matches tmp but not temp.
| | -f configfile | Specify an alternate Aspera configuration file (default is aspera.conf). | | -W token_string | Specify the token string for the transfer. | | -@[range_low:range_high] | Transfer only part of a file. This option only works when downloading a single file and does not support resuming. The argument to “-@” may omit either or both numbers, and the “:” delimiter. For example, -@3000:6000 transfers bytes between positions 3000 to 6000; -@1000: transfers from 1000 to the end of the file; and -@:1000 transfers from beginning to 1000. | | -X rexmsg_size | Adjust the maximum size in bytes of a retransmission request. (Max: 1440). | | —mode=mode | Specify the transfer direction, where mode is either send or recv. | | —user=username | The user name to be authenticated by the transfer server.Important: If you do not specify a user name for the transfer, the local username will be authenticated (by default). In the case of a Windows machine and a domain user, the transfer server will strip the domain from the username (e.g. authenticating “Administrator,” rather than “DOMAIN\Administrator”). Thus, you will need to explicitly specify a domain, if applicable to the user. | | —host=hostname | The server’s address. | | —policy=fixed | high | fair | low | The long option of the transfer policy, which overrides deprecated options -Q(QQ) and -U. For a description of policies, see fasp Transfer Policies.Important: If —policy is set on the command line, it will be reflected in _ascp. If —policy is not set, -Q(QQ) and -U will determine the transfer policy and priority. If no related options are specified on the command line, ascp will use the server-side policy setting, which—by default—is “fair.” | | —file-list=filename | Sources in the file-list are inserted as if they appear on the command-line (right after the —file-list=filename option). The file list supports UTF-8 files and input from the stdin through “-“. The sources can exist on either local host or the remote host (in terms of download), but not on both.
- src
- src2
- …
- srcN
Important: Multiple file-lists and file-pair-lists are not supported in a single ascp command. If multiple files are specified, all but the last one will be ignored. In addition, you cannot use the file-list option while also listing file names on the command-line. Only files within the file-list will succeed. | | —file-pair-list=filename | Specify source-destination pairs in a file using the —file-pair-list=FILENAME option. Note that there is no command-line equivalent. In the case of file pair list files, each source and each destination must be separarated by line endings.
- src1
- dst1
- src2
- dst2
- …
- srcN
- dstN
Important: Multiple file-lists and file-pair-lists are not supported in a single ascp command. If mutiple files are specified, all but the the last one will be ignored. In addition, you cannot use the file-pair-list option while also listing file names on the command-line. Only files within the file-list will succeed. | | ——source-prefix=prefix | Add prefix to the beginning of each source path. | | —symbolic-links=method | Specify rule to handle symbolic links. This option takes following values: (Default: follow)
- follow - Follow symbolic links and transfer the linked files.
- copy - Copy only the alias file. If a file with the same name exists on the destination, the symbolic link will not be copied.
- copy+force - Copy only the alias file. If a file with the same name exists on the destination, the symbolic link will replace the file. If the file of the same name on the destination is a symbolic link to a directory, it will not be replaced.
- skip - Skip the symbolic links.
| | —remove-after-transfer | Add this option to remove all source files (excluding the source directory) once the transfer has completed. | | —remove-empty-directories | Add this option to remove an empty source directory once the transfer has completed. | | —skip-special-files | Add this option to skip special files (for example, devices and pipes). | | —file-manifest=output | Generate a list of all transferred files, where output is none or text (Default: none) | | —file-manifest-path=directory | Specify the path to the file manifest.Important: File manifests can only be stored locally. Thus, if you are using S3, or other non-local storage, you must specify a local manifest path. | | —file-manifest-inprogress-suffix=suffix | Specify the suffix of the file manifest’s temporary file. | | —precalculate-job-size | Add this option to calculate total size before transfer. Please note that the server side conf file setting overrides the ascp command line option. | | —overwrite=method | Overwrite files with the same name. This option takes following values (Default: diff):
- always - Always overwrite the file.
- never - Never overwrite the file. However, note that if the parent folder is not empty, its access, modify, and change times may still be updated.
- diff - Overwrite if file is different from the source (i.e., if a complete file exists at the destination (no .aspx file) and is the same as the source file, then leave it unmodified (no change on timestamp/attributes either); otherwise re-transfer the whole source file). Note this policy interacts with the resume policy.
- older - Overwrite if file is older than the source.
Important: When —overwrite=diff, you must also consider the resume policy (-k{0|1|2|3}). If -k0 (or no -k specified), the source and destination files are always deemed to be different, thereby implying always overwrite. If -k1, the source and destination files are compared based on file attributes (currently, just file size). If -k2, the source and destination files are compared based on sparse checksum. If -k3, the source and destination files are compared based on full checksum. | | —file-crypt=crypt | Encrypt or decrypt files. Replace CRYPT with encrypt or decrypt. Passphrase is required. | | —file-checksum=hash | Report checksums for transferred files, where hash is sha1, md5, or none. | | —partial-file-suffix=suffix | Filename extension on the destination computer while the file is being transferred. Once the file has been completely transferred, this filename extension will be removed. (Default: blank)Note: This option only takes effect when it is set on the receiver side. | | —src-base=prefix | Specify the prefix to be stripped off from each source object. The remaining portion of the source path is kept intact at the destination.For example, the “clips” directory on the remote computer contains the following folders and files:/clips/outgoing/file1 /clips/outgoing/folderA/file2 /clips/outgoing/folderB/file3In this case, to transfer all folders and files within the “outgoing” folder (but not the “outgoing” folder, itself), run the following command:$ ascp -d —src-base=/clips/outgoing/ root@10.0.0.1:/clips/outgoing/ /incomingResult: The following folders and files appear in the “incoming” directory at the destination:(docroot)/incoming/file1 (docroot)/incoming/folderA/file2 (docroot)/incoming/folderB/file3Files outside of the source base (for example, /temp/file4) are skipped from transmission and warnings are generated.
Without —src-base
If —src-base is not used, and the source item is a folder, the contents of the folder are transferred, along with the folder itself. For example:$ ascp -d root@10.0.0.1:/clips/outgoing/ /incomingResult:(docroot)/incoming/outgoing/file1 (docroot)/incoming/outgoing/folderA/file2 (docroot)/incoming/outgoing/folderB/file3If —src-base is not used, and the source item is a file, only the file is transferred, not the folders in the file’s path. For example:$ ascp -d root@10.0.0.1:/clips/outgoing/file1 root@10.0.0.1:/clips/outgoing/folderA/file2 /incomingResult:(docroot)/incoming/file1 (docroot)/incoming/file2For further examples, with and without —src-base, see Ascp File Manipulation Examples | | —proxy=proxy_url | Specify the address of the Aspera proxy server. proxy_url takes the form of:dnat[s]://[username]@server:portThe default ports for DNAT and DNATS protocols are 9091 and 9092. | | —preserve-file-owner-uid | (OS X and Linux/UNIX systems only.) Preserve transferred files’ owner information (uid).Note: This option requires the transfer user be authenticated as a superuser. | | —preserve-file-owner-gid | (OS X and Linux/UNIX systems only.) Preserve transferred files’ group information (gid).*Note: This option requires the transfer user be authenticated as a superuser. | | —ignore-host-key | If you’re prompted to accept a host key when connecting to a remote host, ascp ignores the request. | | —check-sshfp=fingerprint | Check against the server SSH host key fingerprint (for example, f74e5de9ed0d62feaf0616ed1e851133c42a0082). | | —apply-local-docroot | Apply the local docroot. This option is equivalent to setting the environment variable ASPERA_SCP_DOCROOT. |

ascp Options for HTTP Fallback

Option Description
-y {0|1} Enable HTTP Fallback transfer server when UDP connection fails. Set to 1 to enable (default: 0).
-j {0|1} Encode all HTTP transfers as JPEG files. Set to 1 to enable (default: 0).
-Y key_file The HTTPS transfer’s key file name.
-I cert_file The HTTPS certificate’s file name.
-t port Specify the port for HTTP Fallback Server.
-x proxy_server Specify the proxy server address used by HTTP Fallback.

从EBI-ENA下载数据

在https://www.ebi.ac.uk/ena/browser/home搜索想要下载的数据，以txt或者josn下载报告。得到如下信息：
study_accession sample_accession experiment_accession run_accession tax_id scientific_name fastq_ftp submitted_ftp sra_ftp
PRJNA438407 SAMN08770611 SRX4043247 SRR7191154 313920 Blumea balsamifera ftp.sra.ebi.ac.uk/vol1/fastq/SRR719/004/SRR7191154/SRR7191154_1.fastq.gz
ftp.sra.ebi.ac.uk/vol1/fastq/SRR719/004/SRR7191154/SRR7191154_2.fastq.gz ftp.sra.ebi.ac.uk/vol1/srr/SRR719/004/SRR7191154
获得sra，fastq文件下载地址，将ftp.sra.ebi.ac.uk/替换成fasp.sra.ebi.ac.uk:，即可下载，参考以下命令
Following are some examples of how Aspera may be used to download ENA data:

  1. ascp -QT -l 300m -P33001 -i path/to/aspera/installation/etc/asperaweb_id_dsa.openssh \
  2. era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/ERR164/ERR164407/ERR164407.fastq.gz \
  3. local/target/directory


下载示例

  1. ascp -QT -l 300m -P33001 -i /home/linxz/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/SRR719/004/SRR7191154/SRR7191154_1.fastq.gz EBI_ENA/