How to Install Software

This document is intended to be a guide to how to install software here at CCS. We have spent a great deal of discussion to come up with this installation scheme. We believe that once you become familiar with it, you will become very comfortable with it and installs should be very easy. With that said, let's dive into this mess.

Terms

architecture

All computers that have compatible binary formats are considered to be of the same architecture. If two machines can run the same compiled program, they're probably the same architecture. The name for the architecture of any machine in CCS can be found by running the `arch' command, or alternative `uname -m' to get the machine type. (Note that these are not always exactly the same.) For example, all of the SPARCs in the college are members of the sun4 architecture. Our other primary architectures at CCS include mips and alpha. If we ever start to assimilate Solaris running, it will be considered a different architechture since it is not really binary compatable with what we consider sun4's.

[collection]

We separate software packages into a few major groupings, called collections. The collections we currently have include:

adm: Administration software.
beta: Software undergoing beta testing.
gnu: The set of software from the GNU project.
unix: General software. (unix is not the best term for this, but we like it better than misc :)
X11R5: X Windows (R5).
X11R6: X Windows (R6).
Xapps: X Applications. (We realize there may be a few collisions between programs that are X11R6 specific, but its easier to deal with these on a case by case rather than trying to have separate collections.)
com: Commercial Applications.

[software]

This is the 'short/simple' name of the software package. (Example: emacs) It should contain no dashes ('-'), as a dashe is used to separate name from version numbers.

[version]

The version is something like 'major.minor', i.e. 3.11

Installing the Software

We currently have some thousand different different software packages installed, so we certainly need to have the method of installating software clearly defined.

Software Packages

A software package is installed in such a way that all of files related to the package are contained within a single directory tree.

The name of that directory is [software]-[version], where we assume that [software] and [version] don't contain any dashes (convert any dashes to underscores, `-' to `_').

For example, emacs version 19.22 would be installed in a directory called emacs-19.22

The software, when configured, should reference the directory name with the version number. This allows one to install a different version of a software package without having collisions between the two versions. This means that the whole compilation process must be done for each new version of software.

In this way, everything related to a software package is located in one directory, namely [software]-[version]. There will be various references to these files, perhaps even copies of them local to machines, but for the most part the entire package is in this one directory.

Software Package Subdirectories

Each package directory should contain a set of standard directories, it varies with each package, but most will need some, if not all of the following:

bin: Executables run by users
lib: Files referenced during run-time, including libraries
man/man??: Manual page directories, containing man?? and cat?? subdirectories Autogenerate whatis files for the man directories.
Automatically link pointers into the help system to these.
doc: Documentation of various types of formats Automatically link pointers into the help system to these files.
etc: Executables never run by users
rc: Program 'resource configuration' files. Not too many programs use a separate directory for this, but it would be nice if they could be adapted to. It greatly aids in administration if we know where to look. Most programs use etc for such things, but etc is used for programs as well, and we think making the distinction is a win.
include: header files
info: GNU style info files Autogenerate `DIR' files from all of the info files.
Install: Administrator created log files (such as notes on how it was compiled, with what options, etc), SoftInst created log files, and configuration files for SoftInst specific to this package.

Collections

Using just the above scheme, it would be possible for users to get the programs that they want by putting the [package]/bin directories for each software package in their path. However, this is intractable because the path would get intolerably long (too long for some shells) and it would take an enormous amount of maintenance. Therefore we combine many software packages into a collection, defining common directories that unite those packages.

Specifically, a collection of software contains a directory called packages. This directory contains a set of all the software packages in this collection. (All in the form of [software]-[version].)

The software collection directory also has a subdirectory for each of the directories that are known to be in software package directories. Thus it has a bin, a doc, an etc, and so on. Each of these directories is populated with symbolic links pointing to the various package-specific files.

As an example, consider an arbitrary collection of software, it contains the directories which users will reference:

  [collection]/bin
  [collection]/etc
  [collection]/man/man?
  [collection]/lib
  [collection]/packages
  [collection]/src

If emacs-19.25 is installed as part of this collection, then it will be in the packages subdirectory: [collection]/packages/emacs-19.25

The files in the emacs package will be mirrored in the collection directories, so if emacs has these files:

  [collection]/packages/emacs-19.25/bin/emacs
  [collection]/packages/emacs-19.25/bin/etags
  [collection]/packages/emacs-19.25/lib/emacs-lisp

Then these files will be created:

  [collection]/bin/emacs       -> ../packages/emacs-19.25/bin/emacs
  [collection]/bin/etags       -> ../packages/emacs-19.25/bin/etags
  [collection]/lib/emacs-lisp  -> ../packages/emacs-19.25/lib/emacs-lisp

Notice that the symbolic links are made relative, not absolute. This will typically minimize the number of file systems references that will have to be made to locate the file. Further, it allows collections of software packages to be moved as a group without depending on the specific absolute pathname of the collection.

Also notice that the symbolic links refererence the version specific pointer (i.e. "emacs-19.25", not just "emacs"). Although it would seem to make sense to link to "emacs", and then only need to change this one link on a later update, however we have decided that since this is automated by tools, they will be able to resolve uneeded symbolic links. Also, an update to an application does not always contain the same programs/binaries, thus it makes more sense to use the tools to "un-link" the previous version, and "link" the new version.

Source Code

The first thing you proably do when you start to install something is retrieve the source code and start poking around at it.

All source code should be kept in the /ccs/src tree following the format of:

/ccs/src/[category]/[software]-[version].

When you pull down source from somewhere and try to install it, you may find that you need to modify certain files to get it to compile. When you have to modify something, always make a copy of the files to "filename.orig". If you know how to use RCS, it is even better to use, but if you don't, use the ".orig" convention. It really helps to keep these around so we know how the files originally looked.

If you find that you need to make different changes to files on different architectures, you should make backups of the files after they have been modified to work on a certain architecture, i.e. if a Makefile works only on sun4s, make a copy of it called 'Makefile.sun4'.

If you pulled some tricks to get something to compile, it would be useful to others if you document these changes. You should create the file 'Compiling.notes' under the package/Install directory. i.e. for perl, I created /arch/unix/packages/perl-4.036/Install/Compiling.notes which contains the notes on how I got it compiled. In the case of perl, I also realized that the only changes I made were to one file, the config.sh file, so I copied this into that directory as well; this will allow someone to recompile it without figuring out all the configuration options.

You'll be glad you followed these conventions.

Cleaning up the source

Once you have completed the installation (on all of the architechtures), you should be sure to cleanup the src tree.

Specifically, you should do a make clean (or whatever the package uses to clean up) to cleanup the software package, i.e. delete all of the compiled/generate stuff to conserve space. Then you should tar and compress the directory to the filename "[software]-[version].tar.gz". Once this is done, you can remove the directory.

To tar and compress the directory, you could do:

   tar -cvf - software-version | gzip > software-version.tar.gz

or if you use gnu tar, it's simpler:

   tar -cvzf software-version.tar.gz software-version

The Installation Steps

Get the software.
Configure and build it.
Install it.
Test it.
Repeat Step 2 as neccessary for all architectures.
Clean up and verify installation.
Announce it to ccs.systems.announce.

Get the Software

Use programs such as archie and ftp to find and retrieve the software.

You need determine which collection the applications belongs in. If you don't know, you could assume the unix collection and later verify if this is the right collection.

Unpack the source into /ccs/src/[category]/[software]-[version].

If the source is a compressed tar file, you can unpack it like this:

	> cd /ccs/src
	> gzcat foo-1.0.tar.gz | tar xvf -

Warning!! Make sure the tar file will create a directory for all of its files before you attempt to unpack it. (Some misbehaved source packages may just dump all its files right in your current directory.) Check the tar file first like this if you have any doubts:

	> gzcat foo-1.0.tar.gz | tar tf - | head

If the package does not create a subdirectory in the form of [software]-[version], be sure to rename the directory to the proper syntax.

Configure and Build the Software

First, read the documentation that came with the software. That may help answer most of the questions about building the software. Then again, it may not. Potential places to look for documentation are files named README, INSTALL, CHANGES, and directories named man/ or doc/ . A Makefile can be helpful too.

Build (compile) the software according to its own instructions. When configuring the software, specify that it will live in:

/arch/[collection]/packages/[software]-[version]

(with subdirectories bin, man, and so on underneath that).

If the program you're installing is actually just a shell script or some other type of shareable code, then it should be configured to be installed solely in:

/share/[collection]/packages/[software]-[version]

For example, most GNU software can be configured with:

	./configure --prefix=/arch/gnu/packages/[software]-[version]

If the program is not from GNU, how you configure it will vary greatly. You may need to manually edit the Makefile to configure it, it may have it's own configure-like program, or may not have any configuration options at all. It all depends on the package.

If you have to create Makefiles that only work on one kind of architecture, name them `Makefile.[arch]' or follow the conventions that the software package may be using. When building the software, create a symbolic link `Makefile' pointing to the particular Makefile for the architecture you're now building.

If you have to make any custom modifications to the software in order to make it build, documenting your changes in a file called "Compiling.notes", which should be placed under the "Install" directory of the installed application.

Also be sure to keep [file].orig files around if you have to make any custom changes. Alternatively, you can use RCS to track modifications to files.

When you're finished building the software, test it, both as you, and as `testuser'.

Install the software using softinst

softinst is the primary tool you will use to install software. Softinst is designed to be simple to use. It will relieve you of many of the mundane parts of installing software and help insure that software is installed in a standardized manner.

There are essentially four steps to installing software with softinst.

1) Use softinst to define the package and create the standard install directories.: The first thing you need to do is run "softinst [collection]/[software]-[version]". This will ask you a series of questions about the package, answer them to the best of your ability, some of them are optional, some are not.
Try to make your description of the package 'clear' and 'cleanly formatted' as it will be used in an auto-generated software database for users to browse.
2) Install the software into the right directories.: The first step of softinst created the base directories for the install, now you need to use the package's "make install" to install the package into the correct directories.
If the package does have a Makefile, it is good practice to do a "make -n install" to verify that it will do the right thing. If it looks ok, do a "make install" to actually do the install.
Some packages don't automatically install themselves, in this case you would need to manually copy the binaries/scripts into the bin directory, and the manpages into man/man?? directores, etc, etc. Be sure to install everything the "users" need to use the package.
3) Use softinst to clean up the install.: After installing the package pieces into the right directories, you need to run "softinst [collection]/[software]-[version]" again. This will cause softinst to clean up the installation directory. Specifically, it will remove any empty directories, move anything that was considered 'shared' to the /share filesystem, etc.
4) Use softinst once more to complete the installation.: This step could almost be combined with the previous step, but it remains separate so you can verify the install once more. You should check that everything is in the right subdirectories and such before performing this last step.
To complete the last step, run "softinst [collection]/[software]-[version]" again.
This will 'link' the application into the "user" directories of the collection. i.e. it will link [collection]/bin/* -> ../packages/[software]-[version]/bin/*.
The specifics of what it does is configurable (you were able to change the link directectives in the first use of softinst), but for the most part the defaults for each collection are sufficient.

Testing

Your application should now be installed. You'll probably need to "rehash" your shell to get it to notice the new executable(s).

Be sure to thoroughly test the installation before announcing it. Veryfy that all of the files have the correct permissions for all users to read/execute the proper files.

Repeat for all architectures

You should repeat Step 2 as neccessary to install the application on all of the primary architectures.

Clean up the source directory

When the installation is complete, all source files in /ccs/src should be owned by you and have a group of "src" or "beta". All *.o files, executables, and other files which you generated should be removed. (You can usually do this by doing a "make clean".) The original file that you FTP'd or whatever can probably be removed.

If you made minor changes to the source, you may want to #ifdef them into the source. If you made major changes, mail the source back to the author, and tell them to incorporate the changes...

Finally, you should tar and compress the source directory for the package. To tar and compress the directory, you could do:

   tar -cvf - software-version | gzip > software-version.tar.gz

or if you use gnu tar, it's simpler:

   tar -cvzf software-version.tar.gz software-version

Then remove the source directory:

   rm -fr software-version

Announce the new package

Announce the new software by posting news to ccs.systems.announce, if appropriate. The Subject: line should mention the application, the version, and the architecture(s) which it runs on. The message should say a little about what it is, how to use it, and where to get more information. Copying parts of the README files or CHANGES (for upgrades), where appropriate may be useful.

The SoftInst Project