hartman-server/manual.texinfo

\input texinfo
@setfilename manual
@settitle Hartman Lab Server Manual

@copying
Copyright @copyright{} 2021 Bryan C. Roessler
Last updated: @today{}
@end copying

@titlepage
@title Hartman Server Manual
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents

@node Top
@top Lab Server

@menu
* First Chapter::    The first chapter is the
                      only chapter in this sample.
* Index::            Complete index.
@end menu


@node First Chapter
@chapter First Chapter

@cindex chapter, first
This is the first chapter.
@cindex index entry, another

Here is a numbered list.

@enumerate
@item
This is the first item.

@item
This is the second item.
@end enumerate


@node First Section
@section First Section

First section of first chapter.


@node Second Section
@section Second Section

Second section of first chapter.



@node Index
@unnumbered Index

@printindex cp

@bye



Quick Note: At some point in the future UAB may restrict direct ssh access to the Hartman Lab Server. If this occurs you will first need to connect to the UAB intranet via the UAB AnyConnect VPN. Directions for connecting to the UAB VPN can be found here (Linux users, see Fig. A). Once the VPN connection is established you can follow the rest of the manual to connect to the server.

For users that do not have UAB VPN credentials, a whitelist exception will need to be added to the UAB firewall for the specific IP address that the user will be connecting from. The requests can be made here using your UAB credentials.

Type: Permit
Application Name: ssh
Firewall: UAB Internet Border
Source IP Addresses: Users address(es)
Destination IP address: 138.26.17.151
TCP Port: 22
UDP Port: N/A
Other Protocols: N/A
Reason: Outside collaboration/(Other reason)

Server Capabilities

Presently, the server offers the following remote features:

1. SSH/SFTP file server
2. Linux X2Go remote desktop
3. Windows 10 SPICE QEMU/KVM remote desktop
4. Samba-compatible file server
5. Webcam robot monitoring
6. Robot computer remote access

First time login
	
1. Login via ssh client (ssh or putty): ssh blazerid@hartmanlab.genetics.uab.edu
2. Default password is identical to  blazerid
3. System will prompt you to create new password
4. System will log user out after successful password generation
5. Re-login: ssh blazerid@hartmanlab.genetics.uab.edu
6. Enter new password
7. Change smbpasswd: smbpasswd (default password is your blazerid)

SSH/SFTP file server

Files can be  transferred  between systems on the network using the “sftp” command. Use of sftp requires a valid userid and password on the server.

Users can access the server directly through a terminal (text-based) ssh client (ssh in OSX/Linux, or putty in Windows) or can alternatively use a GUI SFTP program such as Filezilla. Linux users can access and mount the SFTP share directly within most file managers (Fig. 2).


A standard user will have access to their $HOME directory as well as /media/data. $HOME resides on a fast (yet storage constrained) SSD while /media/data refers to a large 4TB drive that will be used for lab data and file sharing. Therefore it is recommended that users keep their large files within /media/data and only use $HOME for temporary operations that require fast data access. Presently there is no set limit to user $HOME size but as a general rule users should try to keep their $HOME directories below 30GB of space with the exception of short-term data storage to run calculations. Long-term storage should be kept in /media/data.


Linux X2Go remote desktop

X2Go clients are provided for Windows, OSX, and Linux systems on the X2Go website: http://wiki.x2go.org/doku.php. Linux clients should first check their distribution's repository for the x2goclient software.

While each X2Go client software is slightly unique, the following instructions should provide a general overview of how to connect to the X2Go server (Fig. 3).

Session tab:
Session Name=Hartman Server
Host=hartmanlab.genetics.uab.edu
Login=blazerid
SSH port=22
session type=MATE (required!)

Connection tab:
Connection speed should be set to LAN while the client is connected to the UAB network to provide the best user experience. Compression should be left at the default or be set to “adaptive” for ease of use. While connecting to the server off-campus these quality values can be lowered in order to provide a fluid user experience at the expense of image quality.

Input/output tab:
Set the desired startup window resolution size manually. If there are any issues with keyboard mapping (ex. the arrow keys are not working), select “Configure keyboard” and leave the default settings.

Media tab:
Disable sound support. This will prevent pulseaudio from spamming the server logs

Shared folders:
The user may select folders on his/her local computer to be shared with the server during the active session. The user should browse to the chosen folder on his/her computer, add it to the share, and select “automount”. Upon login, these folders will then appear on the server under /media/disk/<share_name> (Fig. 4).

Once configured, the user can select the session from the right side of X2Go client. If passwordless login is setup, X2Go will automatically login to a new session for that user. If password login is being used, the user will be prompted to enter his/her ssh password before logging in.

Running sessions can be paused or closed from the main client window. Also, multiple session parameters can be stored in the X2Go client, making it easy to select alternate quality settings based on internet client speed, or to provide multiple user login sessions on the same machine.

Note: Some programs do not continue to run at full speed when an X2Go session is paused. In these cases, the program should be run via remote SSH (optionally, in a tmux or screen session) and not in a graphical X2Go session.



Native X forwarding

Linux and OSX: The server supports X11 forwarding so that graphical programs launched via a terminal ssh session will open on the client computer as if the program is running locally. You can activate X11 forwarding by appending the “-X” option to the ssh command: ssh -X blazerid@hartmanlab.genetics.uab.edu

Windows: users can mimic the above functionality by installing Xming and enabling X11 forwarding in the PuTTY options.

You will now be able to launch graphical programs from the server and have them display on your client machine. For example, launch Matlab by running the matlab command after logging into the server via ssh with X forwarding enabled. The Matlab graphical display will launch on your client machine even if Matlab is not installed locally.


Windows SPICE QEMU/KVM remote desktop

A Windows 10 VM is launched automatically on startup and shared with the lab over a SPICE connection. SPICE clients exist for all platforms that will handle the connection seamlessly for the end user. For ease of use it is recommended that Linux,  Windows and OSX users download and install virt-viewer.

The SPICE password for users (port 5900) is: hartmanlab

Alternatively the VMs can be accessed using remote-viewer (Applications >Internet>Remote Viewer) within an existing x2go session (Fig. 5).

The virtualized Windows 10 instances utilize UAB’s Azure protocol for account creation, thus you will need to login with your UAB e-mail address and password. Note: users should NOT login with a pin when prompted—it will disable access to the samba network shares (this is a Windows bug). Users should always log in with a password.

Once you are finished using the Windows virtual machine, it is vital that users log out of their Windows account so that subsequent users do not accidentally login to their account. Windows will perform an automatic logoff after 30 minutes of inactivity.


Robot computer access

While in an x2go or local session, launch remote-viewer (Applications>Internet>Remote Viewer) and create a new connection (connection>new, see Fig. 5):

vnc://192.168.16.101:5900

Samba File Server

While it is possible to mount Linux SFTP shares directly in Windows with third-party tools, it is better to allow native access to files from Windows clients for compatibility reasons. Linux samba shares can be mounted on Windows machines as if the drive exists locally.

Because samba uses a separate protocol from ssh/sftp, it requires a separate authentication token in order to grant users access to the share. On first login, users should change their default samba password using the smbpasswd command. The default password is the same as your blazerid—it is recommended that users select the same password they used for ssh login.

There are two samba shares per user, a private user home directory shared at \\blazerid\blazerid that maps to ~, and a global shared file storage directory shared at \\blazerid\data that maps to /media/data. These directories can be mapped to local drives in the Windows VM and mounted automatically at login (see Fig. 6). 


Webcam robot monitoring

The lab webcam is viewable within an X2Go session by opening a web browser and entering: localhost:8888 in the url bar.

RStudio Server

Newer versions of RStudio do not support IDE access via X2Go. The IDE can be accessed via web browser at localhost:8787 in an X2Go session or via an SSH tunnel, i.e.: 

ssh -f blazerid@hartmanlab.genetics.uab.edu -L 8787:localhost:8787 -N

Backing up data

The rsync, rsnapshot, and syncthing tools for making backups are all pre-installed on the server.

rsync is generally recommended for users that would like to periodically backup their ~ directory to a local machine over ssh. This can be accomplished by running:

rsync -azH –-delete blazerid@hartmanlab.genetics.uab.edu:~/ ~/backup

A graphical alternative is syncthing (Applications>Internet>Syncthing) which runs in the browser and can sync folders or files between machines using encrypted relays.






OPTIONAL

Passwordless (public-private key) authentication

In addition to password-based authentication, the SSH server also support public-private key authentication. Not only is this authentication method more secure than passwords, it will streamline the login process so that the user will not need to enter his/her password for most operations on the server. This is especially convenient if the user regularly transfers files via SCP or SFTP or accesses the remote desktop or VMs.

In order to set up public-private key authentication, the user will need to generate a public and private keypair on the client machine. The user will then store the private key locally on their computer and transfer the public key to the server.

Linux and OSX users can use ssh-keygen and Windows users can use PuTTYgen to generate a public-private keypair. The following is an example using ssh-keygen (note: entering a passphrase is not necessary):

	blazerid@local-host$ ssh-keygen
	Generating public/private rsa key pair.
	Enter file in which to save the key (/home/jsmith/.ssh/id_rsa):[Enter key]
	Enter passphrase (empty for no passphrase): [Enter key]
	Enter same passphrase again: [Enter key]
	Your identification has been saved in /home/jsmith/.ssh/id_rsa.
	Your public key has been saved in /home/jsmith/.ssh/id_rsa.pub.

Linux users can then use the ssh-copy-id command to automatically copy the public key to the server:

	ssh-copy-id -i ~/.ssh/id_rsa.pub blazerid@hartmanlab.genetics.uab.edu


Similar utilities exist for OSX and Windows users but most find it easier to copy and paste the entire contents of the generated id_rsa.pub file into the ~/.ssh/authorizedkeys file on the server (or create the file if it does not yet exist) (Fig. 7).


Linux users have the added benefit that most programs should prioritize public/private key authentication over password based so little additional configuration is necessary.

Once the public key is stored on the server and the user has tested passwordless login via SSH, the X2Go client can be configured to utilize passwordless login by checking the “Try autologin” box (Fig. 3). Windows and OSX users should leave the box disabled and select their private key manually (i.e. id_rsa) in the “Use RSA/DSA key for ssh connection”. Additionally, they may need to add their private key to PuTTY and/or FileZilla/WinSCP to achieve passwordless login.

Once configured, the user will no longer need to enter their password to access the SFTP or X2Go server, which simplifies the process and enhances security.



Additional information for administrators

Installing software
Most software should be available directly from the CentOS repository. The Hartman server also has enabled the EPEL repository which expands upon the software selection available from CentOS. If a user needs a specific piece of software, an administrator should install it for them.

Search the repository for software: sudo yum search "program_name” (quotes are recommended if using * or ? wildcards)
Install the software: sudo yum install <program_name>

Some software (usually obscure or proprietary) will not be available in the repos. In this case an administrator should install the software package to /usr/local and create links to the pertinent binaries in /usr/local/bin, which is accessible to all users.

Most third-party software packages should come with a relevant INSTALL or README file that will provide instructions on how to compile and install the software. When all else fails, try the following “Big 3” commands:

	1. ./configure
	2. make
	3. sudo make install

Maintaining software
The server uses the yum-cron service to automatically download and install security updates. However, periodically the CentOS developers will release updated software packages with new features that a user may need or request. In this case the package should be updated using: sudo yum update <packagename>

System-wide CentOS updates can be performed by running: sudo yum makecache && sudo yum update
It is generally good practice to restart the server after kernel or systemd updates.

Adding a new UNIX user:
	1. sudo system-config-users
	2. Add user
3. Use blazerid, add full name, and add temp password (make same as blazerid), click OK
	4. Select user>properties>password info>force password change on next login

Allow user access to the Samba share:
	1. sudo system-config-users
	2. Select user blazerid>Properties>Groups>Select “smbgrp”>Click OK
	3. sudo smbpasswd -a blazerid
	4. Enter default password (same as blazerid)
	5. Prompt user to change smbpasswd at next login

Make the new user an administrator:
	1. sudo system-config-users
	2. Select user>properties>groups>add user to 'wheel' group (this gives the user power to use the ‘sudo’ command to run programs as root)

Reset a user password:
	1. sudo passwd blazerid
	2. Enter temp password (same as blazerid)
3. Follow step 4 under “Adding a new user” to force the user to set a new password on next login

Remove an existing user and their home directory
	1. sudo userdel -r username


Searching for currently installed software: 
sudo yum list installed | grep -i "program_name"


Enabling/disabling services (systemd):
Ex. smb (samba sharing)
	1. start automatically: sudo systemctl enable smb.service
	2. do not start automatically: sudo systemctl disable smb.service
	3. start: sudo systemctl start smb.service
	4. stop: sudo systemctl stop smb.service
	5. restart: sudo systemctl restart smb.service
	6. reload unit-file: sudo systemctl daemon-reload
	7. read unit-file: sudo systemctl cat smb.service

Maintaining hardware
Adding a new hard drive and automounting it on startup:
	1. create GPT partion table
	2. format w/ filesystem du jour
	3. find partition UUID: sudo blkid
	4. add to fstab (see existing /media/data line as reference): sudo gedit /etc/fstab

Adding a new virtual machine (VM)
Only system administrators can create and manage virtual machine resources. This is accomplished by running: sudo virt-manager

Use virt-manager to create a new machine and optionally copy an existing Windows .qcow2 disk so that Windows and the virtio drivers do not need to be reinstalled.

In case a new VM is required, the virtio virtualization drivers for QEMU/KVM are stored at /usr/share/virtio-win/ and more information can be found at: https://fedoraproject.org/wiki/Windows_Virtio_Drivers

System administrators should note that Windows 10 will not activate over the UAB network KMS correctly from a VM. Open a cmd window as an administrator and run the following to activate Windows 10 in the VM:

slmgr -skms itis-msls.ad.uab.edu
slmgr -ato

You may also need to manually configure the IPv4 DNS servers to connect to UAB network resources. These are: 138.26.5.2, 138.26.5.66

Virtual machine resources
Memory (but not CPU time!) is reserved for each running VM. Thus it is recommended to assign the least amount of memory possible to each running VM to accomplish specific tasks. The default memory assigned to each VM is 8GB.

CPU performance is complicated but in general it is fine to assign the number of hardware CPUs (sans hyperthreading) available to each VM.

Allow new users to access samba share within the Windows VM (Windows bug workaround)
1. Open C:\Windows\system32\drivers\etc\hosts file and copy contents
2. Open new text document, paste contents of existing hosts file and add appropriate blazerid and server IP line (see existing entries)
3. Save as “hosts” (no extension)
4. Copy new hosts file to C:\Windows\system32\drivers\etc\ (allow it to overwrite existing hosts file)
5. The new user will be able to map/access their samaba shares at \\blazerid\data and \\blazerid\blazerid

Making an existing Windows 10 account user an administrator
1. Login to the PC as the Azure AD user you want to be a local admin. This gets the GUID onto the PC.
2. Log out as that user and login as a local admin use
3. Open a command prompt as Administrator and using the command line, add the user to the administrators group: net localgroup administrators AzureAD\blazerid@uab.edu /add
4. Log back in as the user and they will be a local admin now

Creating more VM disk space
Example: sudo qemu-img resize /var/lib/libvirt/images/win10-5901.qcow2 +20G
Then add gparted iso (in /media/share/documentation) as boot device and expand working partition)

Security
The server has several security features in place, namely a stateful firewall and fail2ban brute-force blocker. Since all internet-facing services are initiated via ssh or samba, only two incoming ports are required to be open (22), which limits the attack surface. Fail2ban is configured to whitelist the UAB subnet, however repeated failed authentication attempts from off-campus clients will result in a three hour “cool down” period where any access from that IP address will be blocked. In cases of emergency, this can be reset manually by an administrator using the following command: sudo fail2ban-client set sshd unbanip IPADDRESS

If in the future it is discovered that ssh port scans are affecting server network performance, it is recommended that the ssh port be changed to something non-standard and the clients be reconfigured.

Logging
Global logs can be read using: sudo journalctl
This should be your first point of reference for server problems.

Session problems
If users do not exit X2Go sessions and the server is improperly shut down (i.e., sudden power loss), corruption may occur which will prevent future logins. I recommend running x2goterminate-session as the affected user through ssh to help rectify.

Current limitations and future ideas

There is a bug in the Windows PIN handling and SMB shares. There are some registry workarounds, but it’s best to avoid that and hope that Microsoft patches the bug in the future.

Configuration files
I have hard-linked some common file server configuration files in /media/data/documentation for easy administration.

Resources
RHEL documentation: https://access.redhat.com/documentation/en/red-hat-enterprise-linux/
Navigating the Linux command-line: https://www.digitalocean.com/community/tutorials/basic-linux-navigation-and-file-management
yum cheat sheet: https://access.redhat.com/articles/yum-cheat-sheet
systemd cheat sheet: https://access.redhat.com/articles/systemd-cheat-sheet
QEMU/libvirtd: https://wiki.archlinux.org/index.php/QEMU