The only way to connect to the AVIDD cluster from your desktop or laptop is to use the so called ``secure shell'' and related utilities.
Standard telnet and Kerberized telnet connections to the cluster are disabled, as are the Berkeley r-tools, i.e., rsh, rlogin and the like. FTP and Kerberized FTP do not work on the AVIDD cluster either, so you cannot transfer your data to it this way. You will have to use scp instead.
So in this section we're going to learn about using slogin and scp.
Slogin and scp are commonly distributed with Linux. But they are not distributed with Windows and proprietary UNIXes. I don't know about MacOS-X. Most researchers and students working with AVIDD will be either Linux or Windows users anyway. If you're a Linux user, you probably know all about slogin and scp already, so you can skip most of this section and go to chapter 4 right now. The only thing you need to know is that the front-end node on the AVIDD cluster is called bh1.avidd.iu.edu.
If you don't know all about slogin, or if you're a Windows user and do not know anything about it at all, read on.
The best way to get hold of slogin and scp under Windows is to install Cygwin. Cygwin is a full Linux emulator that runs under Windows. In some ways it is even more than Linux, because it provides its users with seamless access to all Windows facilities and then gives them almost all of Linux on top. Cygwin even provides access to NTFS ACLs (the Cygwin commands that manipulate NTFS ACLs are setfacl and getfacl) and to Microsoft Dfs. For example, the commands
$ cd //tqc.iu.edu/dfs/home $ getfacl gustav # file: gustav # owner: gustav # group: ovpit user::rwx group::r-x other:r-x mask:rwx $ cd /cygdrive/c/winntput me in the Microsoft Dfs directory
\\tqc.iu.edu\dfs\homefirst, then I list NTFS/Dfs ACLs on the sub-directory ``gustav'' and finally I go to
Cygwin comes with the secure shell and related utilities, X11 server and applications, GNU compilers, emacs, make, TeX, inetd utilities (you can run your own telnet and ftp servers), PostgreSQL data base, exim mail server and Apache WWW server. It's all there and it'll cost you nothing. If you wanted to buy all this functionality, you'd end up spending thousands of dollars on software.
How do you go about installing Cygwin? It's easy. Go tohttp://www.redhat.com/download/cygwin.htmland press ``Download Now!''. This will download a small file called ``setup.exe''. When you execute this file, you will be guided through various steps, including the download of binaries and documentation, and then unpacking and installation of the whole package on your PC. You will use the same binary to install Cygwin upgrades and patches down the road.
You don't have to download and install the whole lot. It's up to you how much of the package you really need and want to have. But I have installed all of Cygwin on my Windows 2000 PC and use pretty much most of it too. The whole Cygwin will take about 900MB of disk space on your C drive. If you intend to work with IU clusters a lot, if you plan on doing a lot of scientific work and if you want to do it all from your Windows box but without having to go through a double-boot, I recommend that you install Cygwin.
I am one of these weird and perverted individuals who are not very fanatical about UNIX, Linux, Windows, Macs and the rest. To me it's all overpriced trash, unless it's free (like Linux or Cygwin), in which case it is free trash. Quite ruthlessly I'll reach for anything I can get my hands on, to get the job done as quickly and with as little fuss as possible. This usually implies a mixture of commercial software and freeware. Not all commercial software is worth spending money on, but then not all freeware is going to do the job either. You just have to mix and match, weighing your money, skills and requirements in the process.
So, let us assume that you already have Cygwin installed and there is
going to be this little icon on your Windows desktop that looks like a black
C with a green something inside it. Press on this icon and, assuming
that you have configured everything correctly (and this is going to
take some tinkering) you'll get a window that looks rather slyly like
a Linux window (even though it runs under Windows)
bash prompt, e.g.,
gustav@WOODLANDS:../gustav 14:17:11 !516 $My prompt tells me the name of the machine I'm on, the directory I'm in (but not the full path name, just the last segment in the path), the time of the day and the number of the command I am about to issue. But for the sake of brevity I'm going to truncate it just to the name of the machine and the dollar in the examples that follow.
In this course we are going to use IUB and IUPUI clusters. To connect to one or the other, you have to slogin to either avidd-b.iu.edu (this is the Bloomington cluster) or to avidd-i.iu.edu (this is the IUPUI cluster). In both cases, you'll end up in the same home directory, which is going to be mounted on both clusters.
Let us begin with the Bloomington cluster. If this is your first connection, here is what it is going to look like:
WOODLANDS $ slogin avidd-b.iu.edu email@example.com's password: generating ssh file /N/B/gustav/.ssh/id_rsa ... Generating public/private rsa key pair. Created directory '/N/B/gustav/.ssh'. Your identification has been saved in /N/B/gustav/.ssh/id_rsa. Your public key has been saved in /N/B/gustav/.ssh/id_rsa.pub. The key fingerprint is: ed:84:29:8d:22:70:7d:5f:09:eb:c5:3b:ff:54:61:7b gustav@bh1 adding id to ssh file /N/B/gustav/.ssh/authorized_keys [gustav@bh1 gustav]$Observe that once you have made the connection you end up on the host called bh1. To be more precise, avidd-b.iu.edu evaluates to bh1.uits.indiana.edu and avidd-i.iu.edu evaluates to ih1.uits.iupui.edu. These two are called the head nodes of the two respective clusters. They are the nodes to which you connect from the outside in order to submit your AVIDD jobs. Such head nodes are also called front-end nodes.
The head nodes have more than just one network interface. The interfaces that correspond to avidd-i.iu.edu and avidd-b.iu.edu are on the public campus network. They are Gigi interfaces. These nodes can be seen on other networks too. There is a ``cluster'' network there, and a Myrinet network. The latter is used to run MPI jobs and to support GPFS, the General Parallel File System.
In order to make this first connection I had to type my AVIDD password explicitly. This is tedious, especially if you need to make new connections frequently and if you have a complicated password, and it is not very secure either, because the password travels over the network albeit in an encrypted form.
There is a simple way to change this by reconfiguring ssh on your PC and on the AVIDD cluster to work with DSA or RSA keys instead of passwords. This is more secure and more convenient too.
The procedure is as follows.
First you have to generate your own private/public key pair. You do this by calling a program ssh-keygen. This program will generate the keys and it will place them in the .ssh directory in your Cygwin home (which is like Linux home; on my PC I have simply linked it to the Windows' ``My Documents'').
You can generate RSA or DSA keys with ssh-keygen. DSA keys are more secure, so I recommend the latter. Issue the command:
WOODLANDS $ ssh-keygen -t dsaThe command will ask you for the DSA passphrase. The passphrase can be as long as you wish. Here is the first feature that makes the DSA system more secure than UNIX or Linux passwords, which are usually limited to just eight characters or so (if you have more than eight characters in your Linux password, the characters beyond eight may be ignored). Every character in your DSA passphrase matters, e.g., I have 43 characters in one of my favourite passphrases.
The private key will be stored on ~/.ssh/id_dsa and the public key will be stored on ~/.ssh/id_dsa.pub. You can show your public key to the world safely. Without its private partner it's useless. In particular you can transfer it to the AVIDD cluster and append it to the file ~/.ssh/authorized_keys. The easiest way to do this is to copy it from your local laptop window and then paste it into the AVIDD window, in which you are editing authorized_keys. Make sure the pasted text is a single line. It may happen that the copy/paste process inserts newlines in the string. If it does, simply remove them. This is what my public DSA key looks like:
ssh-dss AAAAB3NzaC1kc3MAAACBAKykdA8AG7Vazhia9fI+uKgsyQzQSCK5LhaQy9XwmEk80hJ/Pg3T 4m+yZ1CS93GM2Z2HXEIbCe39piNgg5d+0mhxaRHP48TUZhqX8pgU4vG89o/LqWmUSDAElbnyjL7VHfIl LCZ465dTJezZpAYLz1B+JU20CKjN4y46rzJsMMznAAAAFQDv1+pusBscm1hq0/Gxiz8E7o+eGQAAAIBc 6fDEDraImCtSty124Wi7rEamNDIabswcObhMCm93HrO9VoNoO97A7c7shvsObbdfwUDCMtYSwkaeHB4o VRR/ULL9FWcxzbv3HFw81PTx1CFcyL2+u8e/1d2itpAruTzcs0QZNQldBRjpMpUz52TSD89WV0ZE1Lox 58LKRy4ixAAAAIATq0aL2bbrXu2tK1QuMXYqFHSQUIXMWiQTW7ARJ8mu/EZ92MXvhRYBQYaSXkcq7HHq qhpV3//sGhv28G5gxxFAIynD9xB7UxH44K0F8vl/KmF3Hldn74m3WWhn6+Xz6JOttRRZa7ZGAzrNkwu4 TD3k3y4hyw4M7p/fhftJ8/o0RA== gustav@WOODLANDSAlthough the text printed above is spread over eight lines, it is, in fact, a single line.
Once you have done all this, you can carry out various transactions with the AVIDD cluster as follows.
WOODLANDS $ ssh-agent bash WOODLANDS $ ssh-add Enter passphrase for /home/gustav/.ssh/id_dsa: Identity added: /home/gustav/.ssh/id_dsa (/home/gustav/.ssh/id_dsa) WOODLANDS $ slogin avidd-b.iu.edu [gustav@bh1 gustav]$ ^D Connection to avidd-b.iu.edu closed. WOODLANDS $ ssh avidd-b.iu.edu date Thu Aug 7 15:10:46 EST 2003 WOODLANDS $ ssh avidd-b.iu.edu ls src WOODLANDS $ scp avidd-b.iu.edu:.bashrc bashrc-avidd .bashrc 100% 124 2.3KB/s 00:00 WOODLANDS $Let me explain what happens here. First I have invoked the program ssh-agent and asked it to execute my login shell, bash. The agent is going to hoard my keys and pass them on to any secure shell transactions transparently, i.e, without me having to type them in explicitly over and over. Once the ssh-agent has forked a new agent-supervised shell for me, I have invoked the command ssh-add in order to add my keys to the agent's cache. This is the only time I actually have to type the passphrase. Now I have issued the slogin command and got right to avidd-b.iu.edu without having to type my passphrase. I can also issue ssh commands without having to type the passphrase and check the date or the content of my home directory on the AVIDD cluster. The last command, scp, transfers the content of my .bashrc file on the AVIDD cluster to bashrc-avidd on my local machine. Again, I am not asked for the password or for the passphrase. The ssh agent takes care of this behind my back.
A useful command to execute in the ssh agent supervised shell is
WOODLANDS $ xterm -sb -sl 300 -n avidd -T avidd -e slogin avidd-b.iu.edu &This command brings up an X11 window on your display with a shell running on the AVIDD cluster. Observe that the xterm program runs locally on your PC, not on the cluster. Exiting the AVIDD shell closes the window automatically.
For this to work, you must have X11 server running on your PC. You can use the one that comes with Cygwin. Its latest version is very good, almost as good as quite expensive commercial offerings, although it still has a couple of glitches here and there and may even hang on you after a day-full of activity.
Here is how I use it on my home computer. First
I have copied a file
/usr/X11R6/bin/startxwin.sh to my own private
modified it to look as follows:
#!/bin/sh export DISPLAY=127.0.0.1:0.0 PATH=/usr/X11R6/bin:$PATH rm -f /tmp/.X11-unix/X0 XWin -multiwindow -clipboard & emacs & xclock & exitI have stripped numerous comment lines off this script to make it shorter. The script cleans the X11 socket from the
/tmp/.X11-unixdirectory (the original script tries to remove the whole directory, but this can fail sometimes, especially if other users run X11 and Cygwin programs on the same system too), then invokes the X11 Cygwin server, called
-clipboardoptions. The first option combines X11 and native Windows displays into one, so that X11 applications can be managed the same way Windows applications are managed and so that you don't get a separate X11 root window in the background. The second option lets you copy and paste between Windows and X11 applications. Then I call
xclockand the script exits. If everything works just fine, you should see the Emacs and the Clock windows pop up and you should also see a large X appear in the right corner of the task bar.
I usually run
ssh-agent . This way every
X11 application carries my DSA keys with it. I can invoke these applications
from my Emacs shell.
I connect to the AVIDD cluster using the following command:
$ xterm -sb -sl 300 -e slogin -X avidd-b.iu.edu &The
-Xoption will set up X11 environment for me automatically on the other side. The
DISPLAYover there will be defined in terms of a socket local to the AVIDD head node. This file is readable to you only and it lives in the
/tmpdirectory. Data sent to the socket will be encrypted and transmitted to your Cygwin X11 server, to be displayed on your screen. This is probably the safest way to use X11, because the data streams are going to be encrypted and because you don't enable access to your X11 display to all users on the AVIDD head node.
-X option will work only between OpenSSH on your
and on the AVIDD cluster. It does not work
to, e.g., SSH2 servers on other machines.