Difference between revisions of "NLP"

From UFRC
Jump to navigation Jump to search
Line 32: Line 32:
  
  
*'''spark-nlp:''' See our [[Spark]] help doc to start a Spark cluster. Spark-nlp Python module is available in tensorflow/2.4.1.  
+
*'''spark-nlp:''' See our [[Spark]] help doc to start a Spark cluster. Spark-nlp Python module is available in tensorflow/2.4.1.
 +
 
 +
*'''parlai:''' Conversational AI framework by Facebook, includes a wide variety of models from 110M to 9B parameters. 
  
 
<!--Configuration-->
 
<!--Configuration-->

Revision as of 18:04, 27 July 2021

Description

This page describes the collection of natural language processing software on HiPerGator.


Environment Modules for NLP

  • nlp: module load nlp will provide a Python environment with spacy, pytorch, torchtext, nltk, sentencepiece, transformers, sentence-transformers, bertopic, RAPIDSai, gensim, scikit-learn, and more.


  • nemo: module load nemo will provide a singularity container environment with Python and Nvidia NeMo. NeMo has NLP task training, plus speech-to-text and text-to-speech models.


  • pytorch: Note, use module spider pytorch to list the version we have available. Beyond stock pytorch versions, we have the Nvidia pytorch singularity container with the Apex optimizers required for Megatron-LM. Use module load ngc-pytorch to access this container, and you can run Megatron from source code.


  • spark-nlp: See our Spark help doc to start a Spark cluster. Spark-nlp Python module is available in tensorflow/2.4.1.
  • parlai: Conversational AI framework by Facebook, includes a wide variety of models from 110M to 9B parameters.