diff --git a/source/cloud-usage.rst b/source/cloud-usage.rst new file mode 100644 index 0000000..2e25106 --- /dev/null +++ b/source/cloud-usage.rst @@ -0,0 +1,203 @@ +Cloud Usage +=========== + +This page covers how the lab uses cloud-based tools and how to properly +access and manage shared resources. + +General Access +-------------- + +- All computing, whether on-premises or remote, should be done on + `BioHPC `_. +- New users should register for an account at + `BioHPC account registration `_. +- New users are required to attend mandatory BioHPC training on the first + Wednesday of every month. BioHPC accounts cannot be activated without + training attendance. +- BioHPC access is provided mainly through + `web based visualization `_. +- It is recommended to download a + `VNC client `_ for WebGUI + access. +- Remote BioHPC access requires VPN access through Global Protect. Information + on how to set up remote access can be found + `here `_. + +Filesystem Navigation +--------------------- + +These are fundamental operations for moving and navigating to data on BioHPC +nodes. + +- Print Working Directory (pwd): Displays the full path of the directory you + are currently in, to determine your location within the filesystem. + +.. code-block:: bash + + pwd + +- List Directory Contents (ls): Shows the files and folders in your current + directory. Use ls -l for detailed information or ls -a to see hidden files. + +.. code-block:: bash + + ls -la + +- Change Directory (cd): Navigates to a different directory. Use cd - to go + to the previous directory, cd ~ to go to your home directory, or provide a + path to navigate to. + +.. code-block:: bash + + cd /path/to/data/ + +- Move or Rename Data (mv): Moves files or directories from one location to + another. This same command is also used to rename files. + +.. code-block:: bash + + mv data_file.csv /new/location/ + mv old_name.txt new_name.txt + +- Copy Data (cp): Duplicates files. If you need to copy an entire directory + and all of its contents, be sure to use the recursive flag (-r). + +.. code-block:: bash + + cp source_file.txt destination_file.txt + cp source_file.txt /destination/data_folder/ + cp -r /source/data_folder/ /destination/data_folder/ + + +Data Processing +--------------- + +Microscopy data can now be processed through +`Astrocyte `_. An existing BioHPC account +is required for Astrocyte access. Astrocyte accounts also share the same +credentials as BioHPC accounts. + +Project Creation +^^^^^^^^^^^^^^^^ + +Workflows can only be run through the usage of Astrocyte projects. Use these +steps to create a project in Astrocyte and upload workflow input data. + +- Go to the My Project page in Astrocyte. + +.. image:: images/astrocyte-my-project-page.png + :alt: Screenshot: Astrocyte My Project page + +- Scroll to the create project section. +- Enter a project name, then press Create. + +.. image:: images/astrocyte-create-project-section.png + :alt: Screenshot: Create project section with project name and Create button + +- After the project opens, find the Input area and click Upload + +.. image:: images/astrocyte-project-input-upload.png + :alt: Screenshot: Upload button under the project Input area. + +- Upload the input data through the upload channel you want to use. + +Data Staging +^^^^^^^^^^^^ + +Data can be uploaded to an Astrocyte project through a variety of provided +channels. If you have not yet created a project, refer to the steps above. + +- Data can be directly uploaded through the WebGUI. + +.. image:: images/direct-upload.png + :alt: Screenshot: Astrocyte direct file upload + +- Data can be copied directly from the BioHPC portal via command line. + +.. image:: images/cli-upload.png + :alt: Screenshot: CLI upload instructions + +- Data can be uploaded through Lamella via share link. + +.. image:: images/lamella-upload.png + :alt: Screenshot: Astrocyte Lamella file upload + + +Provided Workflows +^^^^^^^^^^^^^^^^^^ + +All workflows can be found at the +`lab page `_. View +each workflow's attached documentation for additional information. + +- 3D GPU Deskew Workflow: This workflow normalizes selected ctASLM/light-sheet + microscopy image files to OME-Zarr and runs GPU-accelerated shear/rotation + operations. +- 3D GPU Deconvolution Workflow: This workflow normalizes selected microscopy + image volumes to OME-Zarr, estimates a blind PSF, and runs GPU-accelerated + Richardson-Lucy deconvolution. +- Neuroglancer Visualization: This workflow visualizes 3D OME-Zarr volumes. + +Resource Management +------------------- + +Both BioHPC and Astrocyte bill for compute resource usage based on the time +and type of nodes allocated. To ensure efficient use of your allocations and +to avoid unnecessary charges, always try to keep your resource usage to a +minimum. For information on BioHPC data management, view :doc:`data-management`. + +Generic Queues +^^^^^^^^^^^^^^ + +If you do not require a specific hardware configuration, you can use the +generic queues to get your jobs running efficiently: + +- super: Generic high-performance CPU node. +- GPU: Generic GPU node. + +These generic queues act as a sliding scale for resources. They will +automatically attempt to allocate the weakest available node first. If those +are fully occupied, the system will look to the next strongest node in the +queue (e.g., if all 128GB nodes are busy, the queue will automatically try to +place your job on an available 256GB node). + +Available BioHPC Nodes +^^^^^^^^^^^^^^^^^^^^^^ + +Below is a simplified overview of the available CPU and GPU nodes and what +they are best suited for. + +CPU Nodes +""""""""" + +- 128GB: Entry-level memory node, best for light data processing and basic + analysis. +- 256GB / 256GBv1 / 256GBv2: Standard mid-tier memory nodes, ideal for + everyday bioinformatic workflows and average datasets. +- 384GB: High-memory node, meant for memory-heavy jobs that exceed standard + system limits. +- 512GB: Maximum-capacity memory node, reserved for massive datasets and + extreme processing tasks. + +GPU Nodes +""""""""" + +- GPU2H200: Ultra-high-performance node with next-generation H200 GPUs for + maximum computation speed and massive AI or image processing workloads. +- GPU4A100 / GPU4H100: Heavy-duty multi-GPU nodes with four top-tier cards, + ideal for intensive parallel computing and large-scale deep learning models. +- GPU4v100: Multi-GPU node with four previous-generation V100 cards for + workflows optimized across multiple GPUs. +- GPUA100 / GPUL4 / GPURTX6k: Modern single-GPU nodes ranging from lightweight + acceleration (L4) to high-memory image processing and heavy graphics + workflows (A100, RTX 6000). +- GPUp100 / GPUp4 / GPUp40 / GPUv100s: Standard single and dual GPU nodes + suitable for everyday accelerated tasks, smaller image sets, and general + GPU-enabled pipelines. + +Related pages +------------- + +- :doc:`digital-tools` for account and platform setup +- :doc:`data-management` for information on handling BioHPC data +- :doc:`policies` for lab expectations that intersect with record keeping diff --git a/source/data-management.rst b/source/data-management.rst index c6f8114..82e894c 100644 --- a/source/data-management.rst +++ b/source/data-management.rst @@ -12,6 +12,7 @@ Data Storage `BioHPC account registration `_. - BioHPC access is a standard requirement for working in the lab, and new users are typically directed to an orientation or training session during setup. + See :doc:`cloud-usage` if this information has not yet been provided. - The primary archive location for lab data is ``/archive``. - Organize data routinely and remove material that no longer needs to be kept; storage has a real operational cost. @@ -62,6 +63,7 @@ Brightness and contrast Related pages ------------- +- :doc:`cloud-usage` for BioHPC access, Astrocyte uploads, and compute resource expectations - :doc:`digital-tools` for account and platform setup - :doc:`figure-preparation` for manuscript-ready figure formatting expectations - :doc:`policies` for lab expectations that intersect with record keeping diff --git a/source/departure.rst b/source/departure.rst index 35aceac..742a8ba 100644 --- a/source/departure.rst +++ b/source/departure.rst @@ -11,6 +11,8 @@ Before Your Last Day - Make sure protocols and key procedures are documented in the appropriate shared system. - Organize published data in ``/archive/MIL/publications/`` using a clear project naming scheme that others can navigate later. +- Confirm BioHPC and Astrocyte project materials follow :doc:`cloud-usage` + expectations before handing off cloud-based workflows. - Remove personal information from lab-managed computers and return any lab-owned devices that were assigned to you. - Return keys, badges, laptops, and other UTSW property as instructed. diff --git a/source/digital-tools.rst b/source/digital-tools.rst index 7d40741..35f6bb4 100644 --- a/source/digital-tools.rst +++ b/source/digital-tools.rst @@ -62,10 +62,21 @@ BioHPC - Register during onboarding and make sure you understand the storage structure used by your projects. - See :doc:`data-management` for storage expectations. +- See :doc:`cloud-usage` for access, remote login, Astrocyte, and compute usage expectations. + +Astrocyte +--------- + +- Astrocyte is one of the channels offered by the lab for data analysis. +- An existing BioHPC account is required for Astrocyte access. Login and + registration are the same as your BioHPC credentials. +- See :doc:`data-management` for storage expectations. +- See :doc:`cloud-usage` for project setup, upload options, provided workflows, and resource management. Related Pages ------------- - :doc:`onboarding` +- :doc:`cloud-usage` - :doc:`data-management` - :doc:`working-at-utsw` diff --git a/source/images/astrocyte-create-project-section.png b/source/images/astrocyte-create-project-section.png new file mode 100644 index 0000000..d6706cb Binary files /dev/null and b/source/images/astrocyte-create-project-section.png differ diff --git a/source/images/astrocyte-my-project-page.png b/source/images/astrocyte-my-project-page.png new file mode 100644 index 0000000..d61f995 Binary files /dev/null and b/source/images/astrocyte-my-project-page.png differ diff --git a/source/images/astrocyte-project-input-upload.png b/source/images/astrocyte-project-input-upload.png new file mode 100644 index 0000000..638f9ba Binary files /dev/null and b/source/images/astrocyte-project-input-upload.png differ diff --git a/source/images/cli-upload.png b/source/images/cli-upload.png new file mode 100644 index 0000000..7d57ecd Binary files /dev/null and b/source/images/cli-upload.png differ diff --git a/source/images/direct-upload.png b/source/images/direct-upload.png new file mode 100644 index 0000000..67f15f6 Binary files /dev/null and b/source/images/direct-upload.png differ diff --git a/source/images/lamella-upload.png b/source/images/lamella-upload.png new file mode 100644 index 0000000..5ab963f Binary files /dev/null and b/source/images/lamella-upload.png differ diff --git a/source/index.rst b/source/index.rst index 8b7a50a..137ac36 100644 --- a/source/index.rst +++ b/source/index.rst @@ -23,7 +23,7 @@ Explore the site through the public resources below. Recent work and publication resources. :doc:`Lab Resources ` - Onboarding, policies, protocols, equipment, and data management. + Onboarding, policies, protocols, equipment, cloud usage, and data management. .. toctree:: :caption: Contents diff --git a/source/onboarding.rst b/source/onboarding.rst index dbe8bb6..dca9145 100644 --- a/source/onboarding.rst +++ b/source/onboarding.rst @@ -34,7 +34,9 @@ Computing and records - LabArchives is used for digital lab notebooks and shared lab information. - BioHPC is the primary storage and compute environment for lab data. -See :doc:`digital-tools` for the operational details of these systems. +See :doc:`digital-tools` for the operational details of these systems and +:doc:`cloud-usage` for BioHPC training, VPN access, Astrocyte, and compute +resource expectations. Contacts -------- @@ -103,4 +105,5 @@ Quick reference - Department IT: Hector.EncarnacionPetersen@UTSouthwestern.edu - Badge access: BadgeAccessControl@UTSouthwestern.edu - Core lab systems: see :doc:`digital-tools` +- BioHPC, Astrocyte, and remote compute: see :doc:`cloud-usage` - Institutional logistics: see :doc:`working-at-utsw` diff --git a/source/resources.rst b/source/resources.rst index 764ea46..23d9838 100644 --- a/source/resources.rst +++ b/source/resources.rst @@ -19,6 +19,7 @@ Overview - :doc:`meetings` for recurring meetings, attendance expectations, and presentation formats. - :doc:`digital-tools` for email, Slack, Outlook, GitHub, LabArchives, and BioHPC. - :doc:`working-at-utsw` for institution-facing logistics such as HR, IR, VPN, shipping, and travel. +- :doc:`cloud-usage` for BioHPC, Astrocyte, remote access, data staging, and compute resource expectations. - :doc:`data-management` for storage expectations and microscopy image best practices. - :doc:`figure-preparation` for manuscript-ready figure expectations and formatting standards. - :doc:`protocols` for standard operating procedures and shared workflows. @@ -33,6 +34,7 @@ Overview meetings digital-tools working-at-utsw + cloud-usage data-management figure-preparation equipment diff --git a/source/working-at-utsw.rst b/source/working-at-utsw.rst index 9025149..7c2105f 100644 --- a/source/working-at-utsw.rst +++ b/source/working-at-utsw.rst @@ -55,6 +55,7 @@ VPN and Remote Access - If you need remote access, set up the UTSW VPN while you are on campus if possible. - Limited intranet access is also available through the `Citrix Gateway `_. +- BioHPC and Astrocyte remote workflows are covered in :doc:`cloud-usage`. Equipment Surplus and Decontamination ------------------------------------- diff --git a/tests/test_cloud_usage_navigation.py b/tests/test_cloud_usage_navigation.py new file mode 100644 index 0000000..b9001c4 --- /dev/null +++ b/tests/test_cloud_usage_navigation.py @@ -0,0 +1,33 @@ +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +SOURCE = ROOT / "source" + + +def read_doc(name): + return (SOURCE / name).read_text(encoding="utf-8") + + +def test_cloud_usage_is_owned_by_lab_resources_navigation_only(): + index = read_doc("index.rst") + resources = read_doc("resources.rst") + + assert ":doc:`Cloud Usage `" not in index + assert "\n cloud-usage\n" not in index + assert ":doc:`cloud-usage`" in resources + assert "\n cloud-usage\n" in resources + + +def test_cloud_usage_is_linked_from_resource_paths(): + expected_links = { + "resources.rst": ":doc:`cloud-usage`", + "onboarding.rst": ":doc:`cloud-usage`", + "working-at-utsw.rst": ":doc:`cloud-usage`", + "digital-tools.rst": ":doc:`cloud-usage`", + "data-management.rst": ":doc:`cloud-usage`", + "departure.rst": ":doc:`cloud-usage`", + } + + for filename, link in expected_links.items(): + assert link in read_doc(filename)