Globus
Globus is an easy-to-use, high-performance data transfer tool developed by the Computation Institute, the University of Chicago and Argonne National Laboratory. UF Research Computing has deployed Globus as one mechanism to facilitate data transfer to and from HiPerGator.
Globus uses a grid-FTP network which uses the power of multiple servers to simultaneously transfer data.
UFRC maintains a managed Globus setup with multiple servers for the highest available bandwidth and filesystem throughput. The HPG managed endpoint is 'UFRC HiperGator'
Getting Started
Getting a Globus Account
Globus will redirect you to UF GatorLink authentication when you log into Globus.org. If you created your globus.org account before Globus started using the new authentication you'll need to link your old globus @globusid.org account with your @ufl.edu account.
Sharing Data: Quick Start
If you just want to know how to share a directory on one of the HPG filesystems here is a very brief procedure:
- Go to https://globus.org and log in with your GatorLink credentials after choosing 'University of Florida' as the authentication provider.
- Type or paste 'UFRC HiPerGator' into the collection search box.
- Type or paste the shortest path to the directory tree you want to be in E.g. '/blue/mygroup' or '/orange/mygroup' or the full path to the directory you want to share. You may have to give the 'Globus Application' permission to access files in your browser at this point.
- Browse into the directory you want to share if you didn't paste the full path above.
- Click on the 'Share' menu item.
- Give the share a name and click on 'Create Collection'. A 'Shared Collection' is now set up, but nobody except you has access, yet! If you're having issues creating a collection, use the following link to proceed: Create New Collection.
Note: Everyone you want to give permission via globus has to log into Globus first, so their Globus account would get created and they could tell you their globus identity (username).
- To give globus users permissions to read or write to that directory click on 'Permissions' > 'Add Permissions - Share With' icon and search for or type/paste a globus username.
- Choose Read, Write, or both for the permission as appropriate.
- Add a message that will be sent to the target user by email if you want.
- Click on 'Add Permission' button.
Do not change the path or switch from 'user - share with specific individuals'. If there is a 'Globus' group (we have to set up one if you really need multiple users in a single group) in existence then you can use a group. Otherwise, do the sharing with particular globus users, which are not necessarily HiPerGator users, so don't confuse Globus users/groups with HPG users/groups.
That's all!
Transferring Data
For extended instructions on transferring data using Globus, view instructions at Transfer Data with Globus
Introduction
A Globus transfer happens between two endpoints or a source and a target of a transfer. From the point of view of a user transferring data the endpoints are generally called 'Collections'. There are two levels of collections. One level defines a particular globus setup and is asociated with a computing resource such as a server, cluster, storage system, desktop, laptop, or other system globus is installed on. For example HiPerGator Globus set up has a 'Managed Collection' called 'UFRC HiPerGator' that allows you access to HPG filesystems you are authorized to access. An install of Globus Connect Personal on your laptop or desktop, on the other hand, would require setting up a 'Personal Collection' and giving it a name that can be searched for in the globus interface on https://globus.org. Since a personal system is only for your use the 'Personal Collection' is all you set up in a GCP install. However, under a 'Managed Collection' like 'UFRC HiPerGator' every user can set many 'Guest Collection' instances on their filesystem directories. The guest collections become endpoints for data transfers whether for you or for other globus users you give read or write access to.
UF Research Computing Globus Setup
We manage a multi-server high-throughput Globus install on HiPerGator under the umbrella of a managed Globus collection named 'UFRC HiPerGator' that allows you to transfer data right away or to set up any number of guest collections on your directories in /home, /blue, and /orange filesystems. The 'UFRC HiPerGator' collection can be found by searching within the Globus File Explorer interface on https://globus.org.
A valid HiPerGator user account is required to access 'UFRC HiPerGator' collection directly. However, any globus user you authorized to access a guest collection created under 'UFRC HiPerGator' can transfer data to or from that shared collection without having a HPG account as they are identified by their globus ID that you gave a permission to access a guest collection to.
If you do not have a Research Computing account, you may request one here. Please note that your username and password to activate 'UFRC HiPerGator' collection are your GatorLink credentials. At this time Federated Users cannot directly access the 'UFRC HiPerGator' collection. If you are having problems with your password, please submit a support request through the UF Computing Help Desk or visit https://account.it.ufl.edu/ for self-service e.g. to reset your GatorLink password.
Globus Connect Personal
Globus provides users with Globus Connect Personal software they can use to set up a personal collection on their computing device (Windows, MacOS, Linux) for the purpose of performing data transfers from or to that device.
You will be asked to name a personal globus shared collection while instaling GCP. Once created, that collection can be found by name when searching within the Globus File Explorer interface on https://globus.org. Once run, GCP allows you to configure which directories on your personal device it has access to and what kind of access. Initially, only you can transfer data to or from a personal computing device running GCP. However, if you are a 'Globus Plus' user status, which all University of Florida members already have because UFRC has a Globus License, you can give access to your personal collection on your personal computing device running GCP to other Globus users by creating Shared Collections the way you would do it with a guest collection on HPG. Be very careful here to avoid mistakes while giving other people access to files on your personal computing device using this functionality. See Client Plus for a nice write-up on this functionality.
Once GCP is running and configured you will use the Globus File Explorer interface at https://globus.org for making transfers using your personal collection presented by the GCP instance on your device. Think of GCP as a personal 'Globus Server' and https://globus.org as the interface for making transfers.
Automation
You can automate Globus transfers to avoid having to run GCP personally. See https://docs.globus.org/how-to/automate-with-service-account/ for details. Let us know if you run into any issues.
UFRC Globus Group
Note: this step is not needed if you are only creating and using guest collections created under a managed 'UFRC HiPerGator' collection! This is only needed for creating shared collections on your personal computing device running GCP!
To be able to create shared end-points (see below) on your local computer running GCP, if the transfer will be made to a non-managed endpoint e.g. another local computer with a personal end-point and you are not a University of Florida member you need to have Globus Plus User status. To obtain that status please log into the Globus Interface, click on Groups menu at the top and select Search For Groups. Search for 'University of Florida Research Computing' and access request to the group. Once approved, you will have Globus Plus User status when running 'Globus Connect Personal' software on your local computer.
Direct link to the groups form: https://app.globus.org/groups
Globus CLI
If you need to initiate Globus transfers from the command line e.g. as a part of a workflow you may want to use the Globus Command-Line Interface (CLI). However, before you can initiate globus transfers with the Globus CLI on a given computer, you will have to log in to Globus using the globus login command. If you're in an X11 environment a web browser will be started for you to perform authentication. This happens if you're, for example, are in a terminal in OnDemand console or desktop session and run
$ module load globus $ globus login
If you're in a 'headless' environment where GUI programs cannot run use the --no-local-server argument.
Example:
$ globus login --no-local-server Please log into Globus here: --------------------------- https://auth.globus.org/v2/oauth2/authorize?code_challenge=0dlql_6KJbguIncy3-NH 0w8oBOXABa7DqvZ2O2K4bGs&state=_default&redirect_uri=https%3A%2F%2Fauth.globus.o rg%2Fv2%2Fweb%2Fauth-code&prefill_named_grant=login3.stampede2.tacc.utexas.edu& response_type=code&client_id=95fdeba8-fac2-42bd-a357-e068d82ff78e&scope=openid+ profile+email+urn%3Aglobus%3Aauth%3Ascope%3Atransfer.api.globus.org%3Aall&code_ challenge_method=S256&access_type=offline --------------------------- Enter the resulting Authorization Code here:
Copy the URL from the shell and visit that web page in a web browser on any computer. If you are not currently logged in to Globus on that computer, you will be asked to log in at this time. You may log in with your Globus ID or an ID from any organization, such as University of Florida.
You will be taken to a page where an authorization code is displayed. Copy the code to your computer's clipboard.
Paste the authorization code into the shell where you ran the "globus login --no-local-server" command, as a response to its prompt.
Enter the resulting Authorization Code here: 1zIOeE6leaZpnk0Fxfmi1J8UauARmF You have successfully logged in to the Globus CLI as johnqpublic@globusid.org You can always check your current identity with globus whoami Logout of the Globus CLI with globus logout
You will now be able to issue Globus CLI commands on this computer. You can check your Globus login status at any time with the command globus whoami.