updating training instructions

Author: Mitchob

pages/training-development.md

@@ -3,26 +3,21 @@ title: Training development instructions
 description: How to set up and prepare a BioShell training environment for the BioCommons Training Cooperative, including VM configuration, CVMFS use, trainee directory layout, and pre-snapshot requirements.
 ---
 
-> **Draft:** These instructions are under active development. Contact Mitchell for
-> clarification on any steps or requirements — feedback is welcome.
+> **Draft:** These instructions are a work in progress. If anything is unclear, reach out to Mitchell directly. Feedback is very welcome.
 
-This guide is for **training developers** (`tdevNN` users) setting up BioShell training
-environments on virtual machines. You will configure tools, build training materials, and
-prepare a template that is automatically applied to trainee accounts when VMs are provisioned.
+This guide is for **training developers** (`tdevNN` users) who are setting up BioShell training environments for the BioCommons Training Cooperative. You will configure tools, build training materials, and prepare a template that gets automatically applied to trainee accounts when VMs are provisioned.
 
 ---
 
 ## 1. VM setup and access {#vm-setup}
 
 ### Launch a VM instance
 
-VM instances are provisioned by Giorgia or Mitchell. Machines are named with the prefix `D`
-(for dev) followed by a number, for example `D1`.
+VM instances are provisioned by Giorgia or Mitchell. Each dev machine is named with the prefix `D` followed by a number (for example `D1`).
 
 ### Log in via SSH
 
-A username and IP address will be provided by the training team. Your username follows the
-format `tdevNN` where `NN` matches your VM's prefix number.
+The training team will provide you with a username and IP address. Your username follows the format `tdevNN`, where `NN` matches your VM's prefix number.
 
 ```bash
 ssh tdevNN@<IP_Address>
@@ -32,17 +27,15 @@ ssh tdevNN@<IP_Address>
 
 ## 2. Disk budget {#disk-budget}
 
-The base VM template is approximately **12 GB** on a **30 GB** disk, leaving roughly **16 GB**
-of usable space. This must cover:
+The base VM template takes up approximately **12 GB** on a **30 GB** disk, leaving roughly **16 GB** of usable space. That space has to cover:
 
 - Training materials saved in the template directory (`/etc/skel/`)
 - CVMFS-cached container layers and reference data (written to the CVMFS cache on first use)
 - Trainee working files generated during the session
 
-> **Critical:** If the disk fills up during user creation, the `useradd` step will fail and
-> the trainee account will not be created. Keep `/etc/skel/` as lean as possible.
+> **Watch the disk:** If the disk fills up during user creation, the `useradd` step will fail and the trainee account will not be created. Keep `/etc/skel/` as lean as possible.
 
-Check disk availability with:
+Check how much space you have with:
 
 ```bash
 df -h
@@ -59,7 +52,7 @@ tmpfs           5.0M     0  5.0M   0% /run/lock
 tmpfs           197M   16K  197M   1% /run/user/1000
 ```
 
-To recover disk space:
+If you need to recover some disk space, these commands can help:
 
 ```bash
 sudo apt clean                          # Clear apt cache (~640 MB saving)
@@ -72,16 +65,13 @@ sudo journalctl --vacuum-size=50M       # Trim system logs
 
 ## 3. Using CVMFS resources {#cvmfs}
 
-CVMFS provides read-only, on-demand access to reference data and Singularity containers while
-consuming very little cached local disk space. Always prioritise CVMFS resources over local
-copies — they do not count against the disk budget.
+CVMFS gives you read-only, on-demand access to reference data and Singularity containers. Importantly, CVMFS does not count against your disk budget, so you should always prefer CVMFS resources over local copies wherever possible.
 
-For documentation on using data in the CVMFS repository, see the
-[CVMFS reference data guide](https://docs.google.com/document/d/11YKFZjxzpjKPvyqK3SuAX7zUdgB24ziuGtgdvPcYrBs/edit?tab=t.0#heading=h.fbpu5xkpq96d).
+For a full walkthrough of what is available and how to use it, see the [CVMFS and reference data guide](tools).
 
-### Singularity containers via shelley-bio
+### Finding and installing tools with shelley-bio
 
-Use `shelley-bio` to find and install tools from CVMFS-hosted Singularity images:
+Use `shelley-bio` to search for and install tools from CVMFS-hosted Singularity images:
 
 ```bash
 # Search for available tools
@@ -94,86 +84,90 @@ shelley-bio versions <toolname>
 shelley-bio build <toolname>
 ```
 
-Always check whether a required tool or reference is available on CVMFS before considering
-alternatives. Where a CVMFS path exists, point configs and scripts directly at it — do not
-copy resources into the trainee directory.
+### Symlinking reference data from CVMFS {#symlinking}
+
+Rather than copying reference files into the trainee directory, create a symlink that points directly to the CVMFS path. For example:
+
+```bash
+ln -s /cvmfs/data.galaxyproject.org/byhand/CHM13_T2T_v2.0/seq/CHM13_T2T_v2.0.fa CHM13_T2T_v2.0.fa
+```
+
+This creates a symlink in your current directory pointing to the CVMFS source. You can also give it a custom name or place it somewhere else:
+
+```bash
+ln -s /cvmfs/data.galaxyproject.org/byhand/CHM13_T2T_v2.0/seq/CHM13_T2T_v2.0.fa /path/to/destination/my_reference.fa
+```
 
 ---
 
 ## 4. Developing training materials {#developing-materials}
 
-Set up your home directory (`/home/tdevNN`) exactly as you want trainees to experience it.
-Build and test your workflows there, using CVMFS containers and references throughout. The
-goal is a working, self-contained training environment that a trainee could follow from
-start to finish.
+Set up your home directory (`/home/tdevNN`) exactly as you want trainees to experience it. Build and test your workflows there, using CVMFS containers and references throughout. The goal is a working, self-contained environment that a trainee could follow from start to finish.
 
-> **Data size:** When the Australian BioCommons dataset repository is ready it will be the
-> main access point for training data — like CVMFS, data stored there will not count against
-> VM size. Until then, keep data small. Use the minimum input data needed to demonstrate the
-> workflow. If your ideal dataset is too large to copy to every trainee home directory, see
-> [Section 6](#non-cvmfs) for how to pull it at boot time instead.
+> **Keep data small:** When the Australian BioCommons dataset repository is ready, it will be the main access point for training data. Like CVMFS, data stored there will not count against VM size. Until then, use the minimum input data needed to demonstrate the workflow. If your ideal dataset is too large to copy to every trainee home directory, see [Section 6](#non-cvmfs) for how to pull it at boot time instead.
 
 Once everything works end-to-end, your home directory becomes the template.
 
 ---
 
 ## 5. Trainee directory layout {#directory-layout}
 
-Organise your home directory to reflect what trainees should see on login:
+Organise your home directory to reflect what trainees should see when they first log in. A typical layout might look like this:
 
 ```
 ~/
 ├── README.md                  # Introduction and step-by-step instructions
-├── data/                      # Minimal test input data if not available on CVMFS
-├── containers/                # Containers not available on CVMFS (see Section 6)
-├── references/                # References not available on CVMFS (see Section 6)
+├── data/                      # Minimal test input data if not available on CVMFS otherwise symlinked 
+├── containers/                # Containers symlinked unless not available on CVMFS (see Section 6)
+├── references/                # References symlinked unless not available on CVMFS (see Section 6)
 ├── configs/                   # Workflow configuration files
 ├── scripts/                   # Example and helper scripts
 └── results/                   # Empty output directory for trainee use
 ```
 
-Not all directories are needed for every module — include only what is relevant.
+Not every module will need all of these folders. Only include what is relevant.
+
+### Copy your directory into the template
 
-### Check your size before copying
+Before copying, check how large your home directory is:
 
 ```bash
 du -sh /home/<username>/
 ```
 
-Once satisfied, copy your home directory to `/etc/skel/`:
+Once you are happy with the contents, copy everything to `/etc/skel/`:
 
 ```bash
 sudo cp -r /home/<username>/. /etc/skel/
 ```
 
-Replace `<username>` with your actual username (e.g. `tdev01`).
+Replace `<username>` with your actual username (for example `tdev01`).
 
-Review and clean up the template directory:
+Then review what landed in the template and remove anything you do not want:
 
 ```bash
 ls -a /etc/skel/
 ```
 
-Remove any unwanted files. The contents of `/etc/skel/` act as the template and will be
-applied automatically to every trainee's home directory when their account is created.
+The contents of `/etc/skel/` will be copied automatically into every trainee's home directory when their account is created.
+
+Once you are confident the template is correct, you can clean up your own home directory to free space:
 
-Once you are confident the template is correct, delete the contents of your own home
-directory to free space.
+```bash
+rm -rf ~/*
+```
 
-> **Tip:** Ask to have a snapshot taken at key checkpoints if you want a backup before
-> making significant changes.
+> **Tip:** Ask Giorgia or Mitchell to take a snapshot at key checkpoints if you want a backup before making significant changes.
 
 ---
 
 ## 6. If resources are not available on CVMFS {#non-cvmfs}
 
-If a required container or reference is not available on CVMFS and is too large to include
-directly in the trainee directory, it must be fetched at boot time via the VM provisioning
-script. Prepare the required scripts and share with Giorgia or Mitchell.
+If a required container or reference dataset is not on CVMFS and is too large to include directly in the trainee directory, it needs to be fetched at boot time via the VM provisioning script. Prepare the required scripts and share them with Giorgia or Mitchell.
 
 ### Adding a pull step to the provisioning script
 
-Add downloads before the `useradd` block, targeting the relevant folder within `/etc/skel/`:
+Downloads should be added before the `useradd` block, targeting the relevant folder inside `/etc/skel/`:
 
 ```bash
 # Pull a Singularity container into the trainee containers directory
@@ -185,109 +179,103 @@ mkdir -p /home/$USERNAME/references
 wget -q -O /home/$USERNAME/references/<file> <url>
 ```
 
-Resources pulled this way will appear in the correct location within the trainee's home
-directory, consistent with the layout in [Section 5](#directory-layout).
+Resources pulled this way will appear in the correct location in the trainee's home directory, matching the layout described in [Section 5](#directory-layout).
 
-> **Note:** Boot-time pulls add to your disk budget. Factor their size into your 30 GB total.
+> **Note:** Boot-time pulls count toward your disk budget. Factor their size into your 30 GB total.
 
 ---
 
 ## 7. How trainees are provisioned {#provisioning}
 
-When a VM is built from your snapshot (done by Giorgia or Mitchell), the provisioning script
-runs automatically:
+When a VM is built from your snapshot (done by Giorgia or Mitchell), the provisioning script runs automatically:
 
-- The trainee's username will be `userNN` (e.g. `user1`)
-- Their password is deterministically generated from their username, unique per VM
+- The trainee's username will be `userNN` (for example `user1`)
+- Their password is generated deterministically from their username, and is unique per VM
 - The contents of `/etc/skel/` at snapshot time become the trainee's home directory
 - After the user is created, `/etc/skel/` is cleared to reduce VM image size
 
 ---
 
-## 8. Cloud-init datasource check (pre-snapshot requirement) {#cloud-init}
+## 8. Cloud-init datasource check (required before every snapshot) {#cloud-init}
 
-Running `apt upgrade` during VM setup can silently alter cloud-init configuration, causing all
-VMs built from the snapshot to inherit the wrong hostname. The following must be checked and
-corrected before every snapshot.
+Running `apt upgrade` during VM setup can silently change cloud-init configuration, which causes all VMs built from your snapshot to inherit the wrong hostname. Please check and fix this before asking the VM to be snapshot.
 
-### Step 1 — Check for the offending apt file
+### Step 1: Check for the offending apt file
 
 ```bash
 ls /etc/cloud/cloud.cfg.d/
 ```
 
-A working VM should contain only `05_logging.cfg` and `README`. If `90_dpkg.cfg` is present,
-remove it:
+A working VM should only contain `05_logging.cfg` and `README`. If `90_dpkg.cfg` is present, remove it:
 
 ```bash
 sudo rm /etc/cloud/cloud.cfg.d/90_dpkg.cfg
 ```
 
-### Step 2 — Verify the datasource order
+### Step 2: Verify the datasource order
 
 ```bash
 cat /etc/cloud/cloud.cfg | grep -A5 "datasource_list"
 ```
 
-The output should show only `ConfigDrive` and `OpenStack`:
+The output should list only `ConfigDrive` and `OpenStack`:
 
 ```yaml
 datasource_list:
   - ConfigDrive
   - OpenStack
 ```
 
-### Step 3 — Clean cloud-init state
+### Step 3: Clean cloud-init state
 
 ```bash
 sudo cloud-init clean --logs
 ```
 
-> **Important:** `cloud-init clean` deletes `/etc/cloud/ds-identify.cfg`. Recreate it
-> immediately:
+> **Important:** `cloud-init clean` deletes `/etc/cloud/ds-identify.cfg`. You will need to recreate it:
 
 ```bash
 sudo tee /etc/cloud/ds-identify.cfg << 'EOF'
 policy: enabled
 EOF
 ```
 
-Verify it was created correctly:
+Verify it looks right:
 
 ```bash
 cat /etc/cloud/ds-identify.cfg
 ```
 
 Expected output: `policy: enabled`
 
-### Step 4 — Confirm the datasource
+### Step 4: Confirm the datasource
 
 ```bash
 sudo cloud-init init
 sudo cloud-init status --long
 ```
 
-Check the `detail` line in the output — it should read `DataSourceOpenStackLocal` or
-`DataSourceOpenStack`:
+Look at the `detail` line in the output. It should read `DataSourceOpenStackLocal` or `DataSourceOpenStack`:
 
 ```
 detail: DataSourceOpenStackLocal [net,ver=2]
 ```
 
-If it reads `DataSourceNoCloud`, recheck Steps 1–4 and contact Mitchell or Giorgia if the
-problem persists.
+If it reads `DataSourceNoCloud`, work through Steps 1 to 4 again. If it still does not resolve, contact Mitchell or Giorgia.
 
 ---
 
-## 9. Pre-snapshot checklist {#checklist}
+## Pre-snapshot checklist {#checklist}
+
+Run through this before taking any snapshot:
 
 - [ ] Workflow tested end-to-end from a trainee's perspective
-- [ ] All tools use CVMFS Singularity containers via `shelley-bio` where available
-- [ ] All references point to CVMFS paths where available
+- [ ] All tools use CVMFS Singularity containers installed via `shelley-bio` where available
+- [ ] All references point to CVMFS paths or symlinks where available
 - [ ] Input data is as small as practical
 - [ ] `du -sh /home/<username>/` confirms the directory size is acceptable
-- [ ] Any non-CVMFS resources that are too large to bundle are pulled via the provisioning script
+- [ ] Any non-CVMFS resources that are too large to bundle have been prepared as provisioning script pull steps and shared with Giorgia or Mitchell
 - [ ] `sudo cp -r /home/<username>/. /etc/skel/` completed successfully
 - [ ] `/etc/skel/` contents verified before snapshotting
-- [ ] Total estimated disk usage (base ~12 GB + skel + boot-time pulls) comfortably under 30 GB
+- [ ] Total estimated disk usage (base ~12 GB + skel + boot-time pulls) is comfortably under 30 GB
 - [ ] `/etc/cloud/ds-identify.cfg` is present and reads `policy: enabled`