Alpha

README.md

@@ -1,3 +1,290 @@
-# hartmanlab
+# Hartman Lab Server Manual
 
-A collection of scripts and configs for the John Hartman lab server at UAB
+## Copyright 2021 Bryan C. Roessler
+
+Last updated: 2021-10-18
+
+**Quick Note:** At some point in the future UAB may restrict direct ssh access to the Hartman Lab Server. If this occurs you will first need to connect to the UAB intranet via the UAB AnyConnect VPN. Directions for connecting to the UAB VPN can be found [here](https://www.uab.edu/vpn/) (Linux users, see Fig. A). Once the VPN connection is established you can follow the rest of the manual to connect to the server.
+
+For users that do not have UAB VPN credentials, a whitelist exception will need to be added to the UAB firewall for the specific IP address that the user will be connecting from. The requests can be made [here](https://uabprod.service-now.com/service_portal?id=sc_cat_item&sys_id=daf70746374ce3c0daa253b543990e7f) using your UAB credentials:
+
+```(text)
+Type: Permit
+Application Name: ssh
+Firewall: UAB Internet Border
+Source IP Addresses: User address(es)
+Destination IP address: 138.26.17.151
+TCP Port: 22
+UDP Port: N/A
+Other Protocols: N/A
+Reason: Outside collaboration/(Other reason)
+```
+
+### Server Capabilities
+
+
+1. SSH/SFTP file server
+
+2. Linux X2Go remote desktop
+
+3. Windows 10 SPICE QEMU/KVM remote desktop
+
+4. Samba-compatible file server
+
+5. Webcam robot monitoring
+
+6. Robot computer remote access
+
+### First time login
+
+1. Login via ssh client (ssh or putty): `ssh blazerid@hartmanlab.genetics.uab.edu`
+2. Default password has either been preset by an admin or is identical to blazerid
+3. System will prompt you to create new password
+4. System will log user out after successful password generation
+5. Re-login: `ssh blazerid@hartmanlab.genetics.uab.edu`
+6. Enter new password
+7. Change smbpasswd: `smbpasswd` (default password is your blazerid)
+
+## Features
+
+### SSH/SFTP file server
+
+Files can be transferred to/from the server using `sftp`.
+
+Users can access the server directly through a terminal (text-based) ssh client (`ssh` in OSX/Linux, or [`putty`](http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html) in Windows) or can access the storaeg via an SFTP program such as [Filezilla](https://filezilla-project.org/download.php?type=client). Linux users can access and mount the SFTP share directly within most file managers (Fig. 2) or by using `sshfs`.
+
+### X2Go remote desktop
+
+X2Go clients are provided for Windows, OSX, and Linux systems on the [X2Go website](http://wiki.x2go.org/doku.php) or from your package manager.
+
+While each platform's X2Go client software varies, the following should provide an overview of how to connect to the X2Go server (Fig. 3).
+
+#### Session tab:
+
+- Session Name: Hartman Server
+- Host: hartmanlab.genetics.uab.edu
+- Login: *blazerid*
+- SSH port: 22
+- session type: MATE (**required!**)
+
+#### Connection tab
+
+Connection speed should be set to LAN while the client is connected to the UAB network to provide the best user experience. Compression should be left at the default or be set to *adaptive* for ease of use. While connecting to the server off-campus these quality values can be lowered in order to provide a fluid user experience at the expense of image quality.
+
+#### Input/output tab
+
+Set the desired startup window resolution size manually. If there are any issues with keyboard mapping (ex. the arrow keys are not working), select *Configure keyboard* and leave the default settings.
+
+#### Media tab
+
+Disable sound support. This will prevent `pulseaudio` from spamming the server logs
+
+#### Shared folders
+
+The user may select folders on his/her local computer to be shared with the server during the active session. The user should browse to the chosen folder on his/her computer, add it to the share, and select *automount*. Upon login, these folders will then appear on the server under /media/disk/*<share\_name>* (Fig. 4).
+
+Once configured, the user can select the session from the right side of X2Go client. If passwordless login is setup, X2Go will automatically login to a new session for that user. If password login is being used, the user will be prompted to enter his/her ssh password before logging in.
+
+Running sessions can be paused or closed from the main client window. Also, multiple session parameters can be stored in the X2Go client, making it easy to select alternate quality settings based on internet client speed, or to provide multiple user login sessions on the same machine.
+
+Note: Some programs do not continue to run at full speed when an X2Go session is paused. In these cases, the program should be run via remote SSH (optionally, in a [tmux](https://en.wikipedia.org/wiki/Tmux) or screen session) and not in a graphical X2Go session.
+
+### X forwarding
+
+#### Linux, OSX, ChromeOS
+
+The server supports X11 forwarding so that graphical programs launched via a terminal ssh session will open on the client computer as if the program is running locally. You can activate X11 forwarding by appending the “-X” option to the ssh command: `ssh -X blazerid@hartmanlab.genetics.uab.edu`
+
+#### Windows
+
+Users can install Xming and enable X11 forwarding in the PuTTY options.
+
+You will now be able to launch graphical programs from the server and have them display on your client machine. For example, launch Matlab by running the *matlab* command after logging into the server via ssh with X forwarding enabled. The Matlab graphical display will launch on your client machine even if Matlab is not installed locally.
+
+### Windows 10 Virtual Machines
+
+Windows 10 VMs are accessible via SPICE, a remote desktop protocol that is available for all platforms. For ease of use it is recommended that Linux, [Windows](https://virt-manager.org/download/) and [OSX](https://www.spice-space.org/page/OSX_Client) users download and install `virt-viewer` to access the 
+
+The SPICE password for users (port 5900) is: **hartmanlab**
+
+Alternatively the VMs can be accessed using remote-viewer (*Applications>Internet>Remote Viewer*) within an existing x2go session (Fig. 5).
+
+The virtualized Windows 10 instances require logging in with your UAB e-mail address and password. **Note: users should NOT log in with a pin when prompted--it will disable access to the samba network shares (this is a Windows bug). Users should always log in with a password.**
+
+Once you are finished using the Windows virtual machine, it is vital that users **log out of their Windows account** so that subsequent users do not accidentally login to their account. Windows will perform an automatic logoff after 30 minutes of inactivity.
+
+### Robot computer access
+
+While in an x2go or local session, launch remote-viewer (*Applications>Internet>Remote Viewer*) and create a new connection (*Connection>new*, see Fig. 5):`vnc://192.168.16.101:5900`
+
+### Samba File Server
+
+While it is possible to mount Linux SFTP shares directly in Windows with third-party tools, it is better to allow native access to files from Windows clients for compatibility reasons. Linux **samba** shares can be mounted on Windows machines as if the drive exists locally.
+
+Because samba uses a separate protocol from ssh/sftp, it requires a separate authentication token in order to grant users access to the share. On first login, users should change their default samba password using the *smbpasswd* command. The default password is the same as your *blazerid*—it is recommended that users select the same password they used for ssh login.
+
+There are two samba shares per user, a private user home directory shared at \\\\*blazerid*\\*blazerid* that maps to `~`, and a global shared file storage directory shared at \\\\*blazerid\\data* that maps to /media/data. These directories can be mapped to local drives in the Windows VM and mounted automatically at login (see Fig. 6).
+
+### Webcam robot monitoring
+
+The lab webcam is viewable within an X2Go session by opening a web browser and entering: `localhost:8888` in the url bar.
+
+### RStudio Server
+
+Newer versions of RStudio do not support IDE access via X2Go. The IDE can be accessed via web browser at `localhost:8787` in an X2Go session or via an SSH tunnel, ex. `ssh -f blazerid@hartmanlab.genetics.uab.edu -L 8787:localhost:8787 -N`
+
+## Backing up data
+
+The [`rsync`](https://linux.die.net/man/1/rsync), `rsnapshot`, and [`syncthing`](https://syncthing.net/) backup tools are pre-installed on the server.
+
+`rsync` is recommended for users that would just like to periodically backup their `~` directory to a local machine over ssh:
+`rsync -azH --delete blazerid@hartmanlab.genetics.uab.edu:~/* ~/backup/`
+
+A graphical alternative, [`syncthing`](https://syncthing.net/) (*Applications>Internet>Syncthing*) syncs folders oand files between machines automatically and is accessible at `http://localhost:8384`](http://localhost:8384)
+
+## Recommendations
+
+### Passwordless (public-private key) authentication
+
+In addition to password-based authentication, the SSH server also support public-private key authentication. Not only is this authentication method more secure than passwords, it will streamline the login process so that the user will not need to enter his/her password for most operations on the server. This is especially convenient if the user regularly transfers files via SCP or SFTP or accesses the remote desktop or VMs.
+
+In order to set up public-private key authentication, the user will need to generate a public and private keys on the client machine. The user will then store the private key locally on their computer and transfer the public key to the server.
+
+Linux and OSX users can use `ssh-keygen` and Windows users can use PuTTYgen to generate a public-private key pair. The following is an example using ssh-keygen (note: entering a passphrase is not strictly required):
+
+```(sh)
+blazerid@local-host$ ssh-keygen
+Generating public/private rsa key pair.
+Enter file in which to save the key (/home/blazerid/.ssh/id\_rsa):\[Enter key\]
+Enter passphrase (empty for no passphrase): [Enter key]
+Enter same passphrase again: [Enter key]
+Your identification has been saved in /home/blazerid/.ssh/id\_rsa.
+Your public key has been saved in /home/blazerid/.ssh/id\_rsa.pub.
+```
+
+Then use `ssh-copy-id` to copy the public key to the server: `ssh-copy-id -i ~/.ssh/id\_rsa.pub blazerid@hartmanlab.genetics.uab.edu`
+
+Once the public key is stored on the server and the user has tested passwordless login via SSH, the X2Go client can be configured to utilize passwordless login by checking the *Try autologin* box (Fig. 3). Windows and OSX users should leave the box disabled and select their *private* key manually (i.e. id\_rsa) in the “Use RSA/DSA key for ssh connection”. Additionally, they may need to add their private key to PuTTY and/or FileZilla/WinSCP to achieve passwordless login.
+
+Once configured, the user will no longer need to enter their password to access the SFTP or X2Go server, which simplifies the process and enhances security.
+
+## For administrators
+
+### User management
+
+#### Adding a user
+
+##### CLI
+
+- `sudo script-user-add username`
+  - Optionally pass a second argument `password` to create a user's password for them.
+  - `username` can be anything, but ideally a unique string of small capital letters
+
+##### GUI
+
+1. Launch the *Users & Groups* application from the system control panel (or `sudo system-config-users`)
+2. Add User
+   - Add blazerid or other username, add full name, and create temporary password, click OK
+3. Select *User>properties>password info>Force password change on next login*
+4. *Optional:*
+   - Allow user access to shared storage space
+     1. Select *user blazerid>Properties>Groups>Select “smbgrp”>Click OK*
+     2. Set default Samba password: `sudo smbpasswd -a` *`username`*
+   - Make the new user an administrator
+     1. Select *User>properties>groups>Add* *`username`* *to 'wheel' group*
+
+#### Resetting a user password
+
+##### CLI
+
+1. `sudo script-user-reset-password` *`username`*
+   1. By default the password will be the same as the username (unsafe!)
+   2. You can pass an optional second argument *`password`* to set the password manually, or the keyword `reset` to prompt the user to reset their password
+
+##### GUI
+
+1. Follow step 3 under “Adding a user” to force the user to set a new password on next login
+
+#### Removing a user
+
+##### CLI
+
+1. `sudo script-user-remove` *`username`*
+
+
+### Enabling/disabling services
+
+- Start: *sudo systemctl start smb.service*
+- Stop: *sudo systemctl stop smb.service*
+- Start at boot: *sudo systemctl enable smb.service*
+- Do not start at boot: *sudo systemctl disable smb.service*
+- Restart service: *sudo systemctl restart smb.service*
+- Reload services: *sudo systemctl daemon-reload*
+- Read service: *sudo systemctl cat smb.service*
+
+### Adding a drive
+
+1. `sudo scripts-drive-add /dev/sdX`
+   - To determine the drive suffix `lsblk -f`
+
+### Virtual Machines
+
+- Use `virt-manager` to create a new virtual machine
+  - Optionally copy an existing Windows .qcow2 disk so that Windows and the virtio drivers do not need to be reinstalled.
+  - In case a new VM is required, the Windows virtualization drivers ([`virtio`](https://fedoraproject.org/wiki/Windows_Virtio_Drivers)) are located at [`/usr/share/virtio-win/`](/usr/share/virtio-win/).
+
+- Activate Windows using the UAB license in elevated Powershell:
+
+  ```(powershell)
+  slmgr -skms itis-msls.ad.uab.edu
+  slmgr -ato
+  ```
+
+- Add the UAB DNS server(s) (138.26.5.2, 138.26.5.66) to the Windows network config access UAB resources
+
+#### Allow new users to access samba share within the Windows VM (Windows bug workaround)
+
+1. Open `C:\Windows\system32\\drivers\\etc\\hosts` file and copy contents
+2. Open new text document, paste contents of existing hosts file and add appropriate blazerid and server IP line (see existing entries)
+3. Save as “hosts” (no extension)
+4. Copy new hosts file to C:\\Windows\\system32\\drivers\\etc\\ (allow it to overwrite existing hosts file)
+5. The new user will be able to map/access their samaba shares at *\\\\blazerid\\data* and *\\\\blazerid\\blazerid*
+
+#### Making an existing Windows 10 account user an administrator
+
+1. Login to the PC as the Azure AD user you want to be a local admin. This gets the GUID onto the PC.
+2. Log out as that user and login as a local admin
+3. In elevated Powershell, add the user to the administrators group: `net localgroup administrators AzureAD\\blazerid@uab.edu /add`
+4. Log back in as the user and they will be a local admin now
+
+#### Creating more VM disk space
+
+- `sudo qemu-img resize /var/lib/libvirt/images/win10-5901.qcow2 +20G`
+- Then add gparted iso (in /media/share/documentation) as boot device and expand working partition)
+
+### Security
+
+The server has several security features in place, namely a stateful firewall and fail2ban brute-force blocker. Since all internet-facing services are initiated via ssh or samba, only two incoming ports are required to be open (22), which limits the attack surface. Fail2ban is configured to whitelist the UAB subnet, however repeated failed authentication attempts from off-campus clients will result in a three hour “cool down” period where any access from that IP address will be blocked. In cases of emergency, this can be reset manually by an administrator using the following command: `sudo fail2ban-client set sshd unbanip IPADDRESS`
+
+### Logging
+
+Global logs can be read using: `sudo journalctl`. This should be your first point of reference for server problems.
+
+### Troubleshooting
+
+#### X2go sessions
+
+If users do not exit X2Go sessions and the server is improperly shut down (i.e., sudden power loss), corruption may occur which will prevent future logins. Run `x2goterminate-session` as the affected user to help rectify.
+
+### Current bugs
+
+There is a bug in the Windows PIN handling and SMB shares. There are some registry workarounds, but it’s best to avoid that and hope that Microsoft patches the bug in the future.
+
+### Configuration files
+
+I have hard-linked some common file server configuration files in /media/data/documentation for easy administration.
+
+## Resources
+
+- [RHEL documentation](https://access.redhat.com/documentation/en/red-hat-enterprise-linux/)
+- [Navigating the Linux CLI](https://www.digitalocean.com/community/tutorials/basic-linux-navigation-and-file-management)