Difference between revisions of "Galaxy Data Import"

From UFRC
Jump to navigation Jump to search
 
(3 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
Small files residing on your local machine can be easily added to a Galaxy history through the "Get Data>Upload File" tool.  
 
Small files residing on your local machine can be easily added to a Galaxy history through the "Get Data>Upload File" tool.  
  
However, larger files may take too long to upload directly into galaxy via web browser especially if you are using a slow or unstable network connection. Those files need to be transferred to the UFRC and put under '/ufrc/apps/galaxy/incoming/''your@email''' directory.
+
However, larger files may take too long to upload directly into galaxy via web browser especially if you are using a slow or unstable network connection. Those files need to be transferred to the UFRC and put under '/data/apps/galaxy/incoming/''your@email''' directory.
  
Get Data > Upload File' tool shows all files under <code>/ufrc/apps/galaxy/incoming/''your@email''</code> listed in the upload interface. Simply click on the checkboxes in front of the files you want to upload and then click on the 'Execute' button and the upload should commence. Once you uploaded the files please remove them from <code>/ufrc/apps/galaxy/incoming/''your@email''</code>. The above approach should be much simpler and faster than using the Large Data Upload procedure through the Incoming data library and doesn't require any changes. Please give the system at least 15 minutes to fix permissions on the newly uploaded files if they are not readable by the galaxy user.
+
Get Data > Upload File' tool shows all files under <code>/data/apps/galaxy/incoming/''your@email''</code> listed in the upload interface. Simply click on the checkboxes in front of the files you want to upload and then click on the 'Execute' button and the upload should commence. Once you uploaded the files please remove them from <code>/data/apps/galaxy/incoming/''your@email''</code>. The above approach should be much simpler and faster than using the Large Data Upload procedure through the Incoming data library and doesn't require any changes. Please give the system at least 15 minutes to fix permissions on the newly uploaded files if they are not readable by the galaxy user.
 
----
 
----
  
 
Note: The "Incoming" library must only be used to upload data, not for storage. Please delete the uploaded datasets once they've been imported into histories. All datasets and folders in the "Incoming" shared data library may be deleted after a period of 30 days if they weren't cleaned up by the owner.
 
Note: The "Incoming" library must only be used to upload data, not for storage. Please delete the uploaded datasets once they've been imported into histories. All datasets and folders in the "Incoming" shared data library may be deleted after a period of 30 days if they weren't cleaned up by the owner.
  
If the simplified upload procedure is not working then see the step-by-step procedure for using the Incoming data library below. The first part of getting a large dataset into Galaxy is to make it available to the Galaxy
+
==Detailed Procedure==
==Galaxy Username==
+
===Username===
 
* Find out your Galaxy username. Click on the "User" menu and read the "Logged in as user@hostname" line.
 
* Find out your Galaxy username. Click on the "User" menu and read the "Logged in as user@hostname" line.
 
+
* Create your Galaxy incoming directory
==Create your Galaxy incoming directory==
+
If your Galaxy username is "jdoe@ufl.edu" then the incoming directory you need to create is  <code>/data/apps/galaxy/incoming/jdoe@ufl.edu</code>
If your Galaxy username is "jdoe@ufl.edu" then the incoming directory you need to create is  <code>/ufrc/apps/galaxy/incoming/jdoe@ufl.edu</code>
+
  mkdir /data/apps/galaxy/incoming/jdoe@ufl.edu
  mkdir /ufrc/apps/galaxy/incoming/jdoe@ufl.edu
+
===Copying data to the Galaxy incoming directory===
 
+
====SFTP or Linux====
==Copy data to the Galaxy incoming directory==
+
;Note: You can use either command-line or GUI tools like FileZilla to transfer your data to HiPerGator for upload into Galaxy.
;Note: You can use either command-line or GUI tools like [[FileZilla]] to transfer your data to HiPerGator for upload into Galaxy.
+
* Copy your data '''directory''' to your galaxy incoming directory inside the <code>/data/apps/galaxy/incoming</code> directory.
 
+
For instance, if your Galaxy username is "jdoe@ufl.edu" then your data directory must be copied to <code>/data/apps/galaxy/incoming/jdoe@ufl.edu</code>, so in the end the data files will be inside the   
* Copy your data '''directory''' to your galaxy incoming directory inside the <code>/ufrc/apps/galaxy/incoming</code> directory.
+
  <code>/data/apps/galaxy/incoming/jdoe@ufl.edu/data_dir</code>
 
 
For instance, if your Galaxy username is "jdoe@ufl.edu" then your data directory must be copied to <code>/ufrc/apps/galaxy/incoming/jdoe@ufl.edu</code>, so in the end the data files will be inside the   
 
  <code>/ufrc/apps/galaxy/incoming/jdoe@ufl.edu/data_dir</code>
 
 
directory. Substitute your actual Galaxy username and the name of the directory you copied for "jdoe@ufl.edu" and "data_dir".  
 
directory. Substitute your actual Galaxy username and the name of the directory you copied for "jdoe@ufl.edu" and "data_dir".  
 
 
* Make sure the files are readable by setting the correct access mode
 
* Make sure the files are readable by setting the correct access mode
 
 
Set the 755 mode for the directory and 644 for the files either from within the software you used for transferring the files or by logging into HPC and running the following commands:
 
Set the 755 mode for the directory and 644 for the files either from within the software you used for transferring the files or by logging into HPC and running the following commands:
  cd /ufrc/apps/galaxy/incoming/jdoe@ufl.edu
+
  cd /data/apps/galaxy/incoming/jdoe@ufl.edu
 
  chmod 755 data_dir
 
  chmod 755 data_dir
 
  chmod 644 data_dir/*
 
  chmod 644 data_dir/*
 +
 +
====Samba Share====
 +
If you want to mount the incoming galaxy data directory as a windows drive share use the '<code>\\exasmb.rc.ufl.edu\blue\data\apps\galaxy\incoming</code>' path or mount the blue share and brows to data > apps > galaxy > incoming > your@username
  
 
==Notes==
 
==Notes==
 
* You can use scp/sftp or Globus as described in the [[Transfer Data]] help page to copy your data from a local system to your incoming galaxy directory.
 
* You can use scp/sftp or Globus as described in the [[Transfer Data]] help page to copy your data from a local system to your incoming galaxy directory.
 
* The username is your email address. If you don't remember what email address you used for HiPerGator account registration it can always be looked up by clicking on the User item of the Galaxy's main menu.
 
* The username is your email address. If you don't remember what email address you used for HiPerGator account registration it can always be looked up by clicking on the User item of the Galaxy's main menu.
* Your data files should be in a single sub-directory. For instance, if there is a directory called ''test'' that you are transferring to HiPerGator then the full path to the directory with data files of a user ''jdoe@ufl.edu'' will be <code>/ufrc/apps/galaxy/incoming/jdoe@ufl.edu/test</code>
+
* Your data files should be in a single sub-directory. For instance, if there is a directory called ''test'' that you are transferring to HiPerGator then the full path to the directory with data files of a user ''jdoe@ufl.edu'' will be <code>/data/apps/galaxy/incoming/jdoe@ufl.edu/test</code>

Latest revision as of 21:22, 4 August 2020

Introduction

Small files residing on your local machine can be easily added to a Galaxy history through the "Get Data>Upload File" tool.

However, larger files may take too long to upload directly into galaxy via web browser especially if you are using a slow or unstable network connection. Those files need to be transferred to the UFRC and put under '/data/apps/galaxy/incoming/your@email' directory.

Get Data > Upload File' tool shows all files under /data/apps/galaxy/incoming/your@email listed in the upload interface. Simply click on the checkboxes in front of the files you want to upload and then click on the 'Execute' button and the upload should commence. Once you uploaded the files please remove them from /data/apps/galaxy/incoming/your@email. The above approach should be much simpler and faster than using the Large Data Upload procedure through the Incoming data library and doesn't require any changes. Please give the system at least 15 minutes to fix permissions on the newly uploaded files if they are not readable by the galaxy user.


Note: The "Incoming" library must only be used to upload data, not for storage. Please delete the uploaded datasets once they've been imported into histories. All datasets and folders in the "Incoming" shared data library may be deleted after a period of 30 days if they weren't cleaned up by the owner.

Detailed Procedure

Username

  • Find out your Galaxy username. Click on the "User" menu and read the "Logged in as user@hostname" line.
  • Create your Galaxy incoming directory

If your Galaxy username is "jdoe@ufl.edu" then the incoming directory you need to create is /data/apps/galaxy/incoming/jdoe@ufl.edu

mkdir /data/apps/galaxy/incoming/jdoe@ufl.edu

Copying data to the Galaxy incoming directory

SFTP or Linux

Note
You can use either command-line or GUI tools like FileZilla to transfer your data to HiPerGator for upload into Galaxy.
  • Copy your data directory to your galaxy incoming directory inside the /data/apps/galaxy/incoming directory.

For instance, if your Galaxy username is "jdoe@ufl.edu" then your data directory must be copied to /data/apps/galaxy/incoming/jdoe@ufl.edu, so in the end the data files will be inside the

/data/apps/galaxy/incoming/jdoe@ufl.edu/data_dir

directory. Substitute your actual Galaxy username and the name of the directory you copied for "jdoe@ufl.edu" and "data_dir".

  • Make sure the files are readable by setting the correct access mode

Set the 755 mode for the directory and 644 for the files either from within the software you used for transferring the files or by logging into HPC and running the following commands:

cd /data/apps/galaxy/incoming/jdoe@ufl.edu
chmod 755 data_dir
chmod 644 data_dir/*

Samba Share

If you want to mount the incoming galaxy data directory as a windows drive share use the '\\exasmb.rc.ufl.edu\blue\data\apps\galaxy\incoming' path or mount the blue share and brows to data > apps > galaxy > incoming > your@username

Notes

  • You can use scp/sftp or Globus as described in the Transfer Data help page to copy your data from a local system to your incoming galaxy directory.
  • The username is your email address. If you don't remember what email address you used for HiPerGator account registration it can always be looked up by clicking on the User item of the Galaxy's main menu.
  • Your data files should be in a single sub-directory. For instance, if there is a directory called test that you are transferring to HiPerGator then the full path to the directory with data files of a user jdoe@ufl.edu will be /data/apps/galaxy/incoming/jdoe@ufl.edu/test