Difference between revisions of "Galaxy Data Import"
Moskalenko (talk | contribs) |
|||
| (3 intermediate revisions by one other user not shown) | |||
| Line 1: | Line 1: | ||
[[Category:Galaxy]][[Category:Biology]] | [[Category:Galaxy]][[Category:Biology]] | ||
| + | {|align=right | ||
| + | |__TOC__ | ||
| + | |} | ||
=Introduction= | =Introduction= | ||
Small files residing on your local machine can be easily added to a Galaxy history through the "Get Data>Upload File" tool. | Small files residing on your local machine can be easily added to a Galaxy history through the "Get Data>Upload File" tool. | ||
| − | However, larger files may take too long to upload directly into galaxy via web browser especially if you are using a slow or unstable network connection. Those files need to be transferred to the UFRC and put under '/ | + | However, larger files may take too long to upload directly into galaxy via web browser especially if you are using a slow or unstable network connection. Those files need to be transferred to the UFRC and put under '/data/apps/galaxy/incoming/''your@email''' directory. |
| − | Get Data > Upload File' tool shows all files under <code>/ | + | Get Data > Upload File' tool shows all files under <code>/data/apps/galaxy/incoming/''your@email''</code> listed in the upload interface. Simply click on the checkboxes in front of the files you want to upload and then click on the 'Execute' button and the upload should commence. Once you uploaded the files please remove them from <code>/data/apps/galaxy/incoming/''your@email''</code>. The above approach should be much simpler and faster than using the Large Data Upload procedure through the Incoming data library and doesn't require any changes. Please give the system at least 15 minutes to fix permissions on the newly uploaded files if they are not readable by the galaxy user. |
---- | ---- | ||
| Line 11: | Line 14: | ||
==Detailed Procedure== | ==Detailed Procedure== | ||
| + | ===Username=== | ||
* Find out your Galaxy username. Click on the "User" menu and read the "Logged in as user@hostname" line. | * Find out your Galaxy username. Click on the "User" menu and read the "Logged in as user@hostname" line. | ||
* Create your Galaxy incoming directory | * Create your Galaxy incoming directory | ||
| − | If your Galaxy username is "jdoe@ufl.edu" then the incoming directory you need to create is <code>/ | + | If your Galaxy username is "jdoe@ufl.edu" then the incoming directory you need to create is <code>/data/apps/galaxy/incoming/jdoe@ufl.edu</code> |
| − | mkdir / | + | mkdir /data/apps/galaxy/incoming/jdoe@ufl.edu |
| − | + | ===Copying data to the Galaxy incoming directory=== | |
| + | ====SFTP or Linux==== | ||
;Note: You can use either command-line or GUI tools like FileZilla to transfer your data to HiPerGator for upload into Galaxy. | ;Note: You can use either command-line or GUI tools like FileZilla to transfer your data to HiPerGator for upload into Galaxy. | ||
| − | * Copy your data '''directory''' to your galaxy incoming directory inside the <code>/ | + | * Copy your data '''directory''' to your galaxy incoming directory inside the <code>/data/apps/galaxy/incoming</code> directory. |
| − | For instance, if your Galaxy username is "jdoe@ufl.edu" then your data directory must be copied to <code>/ | + | For instance, if your Galaxy username is "jdoe@ufl.edu" then your data directory must be copied to <code>/data/apps/galaxy/incoming/jdoe@ufl.edu</code>, so in the end the data files will be inside the |
| − | <code>/ | + | <code>/data/apps/galaxy/incoming/jdoe@ufl.edu/data_dir</code> |
directory. Substitute your actual Galaxy username and the name of the directory you copied for "jdoe@ufl.edu" and "data_dir". | directory. Substitute your actual Galaxy username and the name of the directory you copied for "jdoe@ufl.edu" and "data_dir". | ||
* Make sure the files are readable by setting the correct access mode | * Make sure the files are readable by setting the correct access mode | ||
Set the 755 mode for the directory and 644 for the files either from within the software you used for transferring the files or by logging into HPC and running the following commands: | Set the 755 mode for the directory and 644 for the files either from within the software you used for transferring the files or by logging into HPC and running the following commands: | ||
| − | cd / | + | cd /data/apps/galaxy/incoming/jdoe@ufl.edu |
chmod 755 data_dir | chmod 755 data_dir | ||
chmod 644 data_dir/* | chmod 644 data_dir/* | ||
| + | |||
| + | ====Samba Share==== | ||
| + | If you want to mount the incoming galaxy data directory as a windows drive share use the '<code>\\exasmb.rc.ufl.edu\blue\data\apps\galaxy\incoming</code>' path or mount the blue share and brows to data > apps > galaxy > incoming > your@username | ||
==Notes== | ==Notes== | ||
* You can use scp/sftp or Globus as described in the [[Transfer Data]] help page to copy your data from a local system to your incoming galaxy directory. | * You can use scp/sftp or Globus as described in the [[Transfer Data]] help page to copy your data from a local system to your incoming galaxy directory. | ||
* The username is your email address. If you don't remember what email address you used for HiPerGator account registration it can always be looked up by clicking on the User item of the Galaxy's main menu. | * The username is your email address. If you don't remember what email address you used for HiPerGator account registration it can always be looked up by clicking on the User item of the Galaxy's main menu. | ||
| − | * Your data files should be in a single sub-directory. For instance, if there is a directory called ''test'' that you are transferring to HiPerGator then the full path to the directory with data files of a user ''jdoe@ufl.edu'' will be <code>/ | + | * Your data files should be in a single sub-directory. For instance, if there is a directory called ''test'' that you are transferring to HiPerGator then the full path to the directory with data files of a user ''jdoe@ufl.edu'' will be <code>/data/apps/galaxy/incoming/jdoe@ufl.edu/test</code> |
Latest revision as of 17:24, 15 December 2022
Introduction
Small files residing on your local machine can be easily added to a Galaxy history through the "Get Data>Upload File" tool.
However, larger files may take too long to upload directly into galaxy via web browser especially if you are using a slow or unstable network connection. Those files need to be transferred to the UFRC and put under '/data/apps/galaxy/incoming/your@email' directory.
Get Data > Upload File' tool shows all files under /data/apps/galaxy/incoming/your@email listed in the upload interface. Simply click on the checkboxes in front of the files you want to upload and then click on the 'Execute' button and the upload should commence. Once you uploaded the files please remove them from /data/apps/galaxy/incoming/your@email. The above approach should be much simpler and faster than using the Large Data Upload procedure through the Incoming data library and doesn't require any changes. Please give the system at least 15 minutes to fix permissions on the newly uploaded files if they are not readable by the galaxy user.
Note: The "Incoming" library must only be used to upload data, not for storage. Please delete the uploaded datasets once they've been imported into histories. All datasets and folders in the "Incoming" shared data library may be deleted after a period of 30 days if they weren't cleaned up by the owner.
Detailed Procedure
Username
- Find out your Galaxy username. Click on the "User" menu and read the "Logged in as user@hostname" line.
- Create your Galaxy incoming directory
If your Galaxy username is "jdoe@ufl.edu" then the incoming directory you need to create is /data/apps/galaxy/incoming/jdoe@ufl.edu
mkdir /data/apps/galaxy/incoming/jdoe@ufl.edu
Copying data to the Galaxy incoming directory
SFTP or Linux
- Note
- You can use either command-line or GUI tools like FileZilla to transfer your data to HiPerGator for upload into Galaxy.
- Copy your data directory to your galaxy incoming directory inside the
/data/apps/galaxy/incomingdirectory.
For instance, if your Galaxy username is "jdoe@ufl.edu" then your data directory must be copied to /data/apps/galaxy/incoming/jdoe@ufl.edu, so in the end the data files will be inside the
/data/apps/galaxy/incoming/jdoe@ufl.edu/data_dir
directory. Substitute your actual Galaxy username and the name of the directory you copied for "jdoe@ufl.edu" and "data_dir".
- Make sure the files are readable by setting the correct access mode
Set the 755 mode for the directory and 644 for the files either from within the software you used for transferring the files or by logging into HPC and running the following commands:
cd /data/apps/galaxy/incoming/jdoe@ufl.edu chmod 755 data_dir chmod 644 data_dir/*
If you want to mount the incoming galaxy data directory as a windows drive share use the '\\exasmb.rc.ufl.edu\blue\data\apps\galaxy\incoming' path or mount the blue share and brows to data > apps > galaxy > incoming > your@username
Notes
- You can use scp/sftp or Globus as described in the Transfer Data help page to copy your data from a local system to your incoming galaxy directory.
- The username is your email address. If you don't remember what email address you used for HiPerGator account registration it can always be looked up by clicking on the User item of the Galaxy's main menu.
- Your data files should be in a single sub-directory. For instance, if there is a directory called test that you are transferring to HiPerGator then the full path to the directory with data files of a user jdoe@ufl.edu will be
/data/apps/galaxy/incoming/jdoe@ufl.edu/test