Difference between revisions of "SRA"

From UFRC
Jump to navigation Jump to search
m (Text replace - "<!--Choose sections to enable - OPTIONAL--> |{{#vardefine:mod|1}} <!--Present instructions for running the software with modules -->" to "")
 
(17 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
__NOTOC__
 
__NOTOC__
 
__NOEDITSECTION__
 
__NOEDITSECTION__
[[Category:Software]]
+
[[Category:Software]][[Category:Biology]][[Category:NGS]]
 
+
{|<!--Main settings - REQUIRED-->
{|
 
<!--Main settings - REQUIRED-->
 
 
|{{#vardefine:app|sra}}
 
|{{#vardefine:app|sra}}
 
|{{#vardefine:url|http://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?cmd=show&f=software&m=software&s=software}}
 
|{{#vardefine:url|http://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?cmd=show&f=software&m=software&s=software}}
<!--Compiler and MPI settings - OPTIONAL -->
+
|{{#vardefine:exe|1}} <!--Present manual instructions for running the software -->
|{{#vardefine:intel|}} <!-- E.g. "11.1" -->
 
|{{#vardefine:mpi|}} <!-- E.g. "openmpi/1.3.4" -->
 
 
 
|{{#vardefine:exe|}} <!--Present manual instructions for running the software -->
 
 
|{{#vardefine:conf|}} <!--Enable config wiki page link - {{#vardefine:conf|1}} = ON/conf|}} = OFF-->
 
|{{#vardefine:conf|}} <!--Enable config wiki page link - {{#vardefine:conf|1}} = ON/conf|}} = OFF-->
 
|{{#vardefine:pbs|}} <!--Enable PBS script wiki page link-->
 
|{{#vardefine:pbs|}} <!--Enable PBS script wiki page link-->
Line 26: Line 20:
 
This is the NCBI Short Read Archive Toolkit.
 
This is the NCBI Short Read Archive Toolkit.
  
''Release notes:''
+
;Note: sra will create a $HOME/ncbi/public directory and cache the prefetched data files there. However, home directory has a 40gb limit and its use for job data storage is a violation of the [https://www.rc.ufl.edu/about/policies/storage/ UFRC storage policy]. You must change that location to a directory in your ufrc space before running the sra toolkit. The official approach is to use the vdb-config tool
 
+
vdb-config -i
SRA Toolkit 2.1.7a includes new features in sam-dump tool and vdb-dump tools.
+
and change the directory to, for example, /blue/$GROUP/$USER/ncbi/public. See the [https://github.com/ncbi/sra-tools/wiki/Toolkit-Configuration SRA Toolkit Configuration Documentation] for more details.
  
Sam-dump now supports slicing across multiple sequences, and dumping
+
Alternatively, create an 'ncbi' directory in your /blue space and symlink it to ~/ncbi. E.g.
cSRA files to fasta and fastq formats. In addition, sam-dump has three
 
new parameters:
 
  
  -=|--hide-identical              Output '=' if base is identical to reference
+
  $ mkdir /blue/mygroup/$USER/ncbi
  --gzip                          Compress output using gzip
+
  $ ln -s /blue/mygroup/$USER/ncbi ~/ncbi
--bzip2                          Compress output using bzip2
 
  
vdb-dump has two new parameters
+
==Uploads==
-o|--column_enum_short          enumerates columns in short form
+
It appears that data uploads to NCBI only work from login servers. Start a screen session before beginning an upload if there are any concerns about being disconnected.
-b|--boolean                    defines how boolean's are printed (1,T)
 
 
 
We have combined the functionality of two scripts, config-assistant.perl and reference-assistant.perl
 
into a single script, configuration-assistant.perl that helps users download the correct references
 
for a given cSRA file and configure the user environment for the SRA Toolkit.
 
<!--Location-->
 
 
<!--Modules-->
 
<!--Modules-->
==Required Modules==
+
==Environment Modules==
[[Modules|modules documentation]]
+
Run <code>module spider {{#var:app}}</code> to find out what environment modules are available for this application.
===Serial===
+
==System Variables==
*{{#var:app}}
+
* HPC_{{uc:{{#var:app}}}}_DIR - installation directory
 
* HPC_SRA_BIN - location of the executables directory
 
* HPC_SRA_BIN - location of the executables directory
 
* HPC_SRA_DOC - location of the documentation directory
 
* HPC_SRA_DOC - location of the documentation directory
 
+
<!--Additional-->
==Aspera Connect==
+
{{#if: {{#var: exe}}|==Additional Information==
 +
===Aspera Connect===
 
To download SRA data you can use the "ascp" utility from the [http://asperasoft.com/downloads/ Aspera Connect] browser plugin package. We have a copy installed and provided by the sra module. A wrapper script ''ascp.sh'' that automatically uses the ssh key is available. For instance:
 
To download SRA data you can use the "ascp" utility from the [http://asperasoft.com/downloads/ Aspera Connect] browser plugin package. We have a copy installed and provided by the sra module. A wrapper script ''ascp.sh'' that automatically uses the ssh key is available. For instance:
 
 
  ascp.sh -QT anonftp@ftp-private.ncbi.nlm.nih.gov:/genomes/Bacteria/all.faa.tar.gz faa
 
  ascp.sh -QT anonftp@ftp-private.ncbi.nlm.nih.gov:/genomes/Bacteria/all.faa.tar.gz faa
 
 
will download the all.faa.tar.gz archive to the faa directory.  
 
will download the all.faa.tar.gz archive to the faa directory.  
  
'''Note:''' if the download fails to start on the first try with a "Session Stop (Error: Client unable to connect to server (check UDP port and firewall))"
+
'''Note:''' if the download fails to run with a "Session Stop (Error: Client unable to connect to server (check UDP port and firewall))"
error just re-run the command. It's a DNS (host name resolution) problem, which will resolve itself.
+
error ''make sure you are running aspera connect on a login node''. If you are already doing the transfer on a login node please submit a support request. This means that the remote site has not been allowed through the firewall. Please be sure to include the path to a script you used to run the data transfer command into the request. Do not put any sensitive information like passwords, keys, and such into the request.
{{#if: {{#var: exe}}|==How To Run==
+
|}}
WRITE INSTRUCTIONS ON RUNNING THE ACTUAL BINARY|}}
 
 
{{#if: {{#var: conf}}|==Configuration==
 
{{#if: {{#var: conf}}|==Configuration==
 
See the [[{{PAGENAME}}_Configuration]] page for {{#var: app}} configuration details.|}}
 
See the [[{{PAGENAME}}_Configuration]] page for {{#var: app}} configuration details.|}}

Latest revision as of 19:10, 27 November 2023

Description

sra website  

This is the NCBI Short Read Archive Toolkit.

Note
sra will create a $HOME/ncbi/public directory and cache the prefetched data files there. However, home directory has a 40gb limit and its use for job data storage is a violation of the UFRC storage policy. You must change that location to a directory in your ufrc space before running the sra toolkit. The official approach is to use the vdb-config tool
vdb-config -i

and change the directory to, for example, /blue/$GROUP/$USER/ncbi/public. See the SRA Toolkit Configuration Documentation for more details.

Alternatively, create an 'ncbi' directory in your /blue space and symlink it to ~/ncbi. E.g.

$ mkdir /blue/mygroup/$USER/ncbi
$ ln -s /blue/mygroup/$USER/ncbi ~/ncbi

Uploads

It appears that data uploads to NCBI only work from login servers. Start a screen session before beginning an upload if there are any concerns about being disconnected.

Environment Modules

Run module spider sra to find out what environment modules are available for this application.

System Variables

  • HPC_SRA_DIR - installation directory
  • HPC_SRA_BIN - location of the executables directory
  • HPC_SRA_DOC - location of the documentation directory

Additional Information

Aspera Connect

To download SRA data you can use the "ascp" utility from the Aspera Connect browser plugin package. We have a copy installed and provided by the sra module. A wrapper script ascp.sh that automatically uses the ssh key is available. For instance:

ascp.sh -QT anonftp@ftp-private.ncbi.nlm.nih.gov:/genomes/Bacteria/all.faa.tar.gz faa

will download the all.faa.tar.gz archive to the faa directory.

Note: if the download fails to run with a "Session Stop (Error: Client unable to connect to server (check UDP port and firewall))" error make sure you are running aspera connect on a login node. If you are already doing the transfer on a login node please submit a support request. This means that the remote site has not been allowed through the firewall. Please be sure to include the path to a script you used to run the data transfer command into the request. Do not put any sensitive information like passwords, keys, and such into the request.