Proposal: NetBSD System Installation Packages
=============================================
CONTENTS
--------
0. Introduction
1. System Packages
1.1 Package Format
1.2 Package Granularity
1.2.1 Root/User/Share separation
2. Package Sets
2.1 Set format
3. Creation of Packages and Sets
4. Modifications to the NetBSD installation process
A. Working Plan
The current NetBSD installation process involves the downloading
of binary `sets', which the user can choose among at install time.
A set is a tarred, gzipped set of files, to be untarred relative
to '/'. No facility exists to choose convenient subsets of the files
in a set to be installed, or to remove a set which has been installed.
The current granularity of sets is very large, being divided into:
base -- general system binaries
comp -- compilers and related tools
etc -- system configuration files
games -- games and other amusements
man -- system manual pages
misc -- items not falling into other categories
secr -- items not exportable under US law
text -- text processing tools
xbase -- general X11R7 binaries
xcomp -- X11R7 development items
xfont -- X11R7 fonts
xserver -- X11R7 servers for various video hardware
Users who wish to install part of a set need to either install
the full set and then determine which files they need to remove,
or abandon the normal install process, and figure out which files
to unpack by hand. Similarly, if a set is later determined to
be unnecessary, the only way to remove it is to figure out which
files on the system belonged to that set, and remove them by hand.
When it comes time to upgrade a system which has been installed this
way, the usual procedure is to unpack a new version of each installed
set over the previous version. When a file is moved, renamed, or
removed in a newer version of a set, the old version often remains on
the system for some time. In at least one recent instance (the move
of /sbin/mountd to /usr/sbin/mountd) this has resulted in much
confusion, and large amounts of traffic on the relevant mailing lists.
The remainder of this document describes a proposed method of handling
these and other problems with the current install set system by
moving to the use of fine-grained `system packages', based on the
currently existing package system for third-party software, and
allowing users to choose among either `package sets' at the same
granularity as our current install sets, or individual `packages'
at a much finer level of granularity. In either case, the new system
would also greatly simplify upgrading or removal of such packages
and sets at a later time, and would allow tracking of dependencies
between the various sets and packages distributed as part of NetBSD.
First, the format of system packages in the proposed system is
discussed, followed by the format of package sets, which will serve
as a replacement for the current install sets. The creation of
packages in an automated fashion from a NetBSD source tree is
discussed as is the effect of this system on the NetBSD installation
process. An appendix discusses my work plan to implement this new
system.
It is hoped that this document will serve as a basis for discussion
of what is involved in changing NetBSD to use system packages for
system installation and upgrades, and that after several iterations
of discussion and revision, it will serve as a plan for the actual
implementation of this system.
System packages will be the basic building blocks of a NetBSD system.
At install time, the user will choose which system packages to install,
subject to dependencies between packages. After system install,
users will be able to install additional packages or remove installed
packages. When it comes time to upgrade the system, packages can
be removed and reinstalled in a reliable fashion. All of this
functionality is already available for third-party software via the
use of the software package system in /usr/pkgsrc. This proposal
extends that functionality to the NetBSD system itself.
1.1 Package Format
System packages will be identical in format to the binary packages
used by the current third-party package system. This will allow the
same tools to be used for working with system packages as are
currently used for working with third-party packages. This will also
also allow the system to benefit from the fact that the workings of
the current package system are well understood.
1.2 Package Granularity
System packages will be at the granularity of groups of related tools
and their support files. Thus, `Kerberos', `UUCP', `Text formatting'
and `amd' might each be packages which depended on nothing but a few
base packages, while `C Development' and `Fortran development' might
be separate packages which each depended upon `Binutils' and `Base
EGCS utilities' packages. Packages sets, described below, would add
the ability to choose entire broad categories of software to install,
like todays install sets, while maintaining the ability to remove
individual packages later.
1.2.1 Root/User/Share separation
In order to support a variety of system configurations, it is crucial
that the new package system support the possibility of some part of
a system residing on a server and possibly being shared between
multiple machines on a network. A machine which has some filesystems
local and some shared must, at the very least, be able to add and
remove packages from local filesystems, and should be able to
determine what packages have been added or removed from the volumes
mounted over the network.
The most common shared configurations are to have a system share
/usr/share from the network, and have all other filesystems local,
or to share the entirety of /usr from the network, and maintain
local root and /var hierarchies, possibly as a single filesystem.
Other commonly shared hierarchies include /usr/X11R7 and /usr/pkg.
Two steps are necessary to support this type of sharing: the system
must be able to check separate repositories for packages installed
on different filesystems, and packages must be designed so as to
allow a client to install only those parts of the system which reside
on local filesystems.
The first of these is addressed by a set of patches described by
Alistair Crooks in a post to the netbsd-current mailing list on
Friday, September 18, 1998. These patches, which have not yet been
committed cause third-party software packages installed in /usr/pkg
to be registered in /usr/pkg/etc/pkg, and packages installed in
/usr/X11R7 to be registered in /usr/X11R7/etc/pkg. This could be
extended easily to allow sharing of system package installations by
having the new system X11R7 packages also use /usr/X11R7/etc/pkg
for package registration, to have system packages installed in /usr
use /usr/etc/pkg for package registration, and to have system
packages installed in / and /var use /etc/pkg for package
registration. This would allow all of the types of filesystem
sharing described above, without introducing too much complication
into the package system.
The second step, that of insuring that a client can choose to install
only the parts of the system which reside on local volumes can be
most easily addressed by careful consideration of package contents.
A look through the contents of the current install sets suggests
that relatively few packages will in fact need to install in more
than one of /, /usr, /usr/share and /usr/X11R7. Were such packages
split into separate components, based on filesystem boundaries,
users would easily be able to install only the parts which are local
in their particular configuration.
In moving to fine-grained system packages, it is important that
beginning users still be able to select broad categories of software
to install at once. The introduction of `package sets', analogous
in granularity, but not mechanism, to the current binary install sets
addresses this concern, while maintaining the ability of more advanced
users to choose among individual packages at install time, and
maintaining the ability to remove, upgrade, or add individual
packages at a later time.
These package sets will maintain the same layout as the current
install sets, so that a user who chooses the same sets as he would
have chosen now will see the same results. In the new system,
however, these sets will be made up of binary packages, and installing
a set will simply result in the installation of the constituent
packages.
2.1 Set format
A set will be a tar archive containing the packages which make up the
set plus a contents file. At the least, the index file will contain
the name of each included package, plus a one line description of each
package's contents. Installation utilities will offer the option of
installing the whole set, or choosing among individual packages,
based on the descriptions in the contents file. It is expected that the
contents file itself will be automatically generated from the one-line
descriptions provided in each package's pkg/COMMENT file.
When a set is installed, the contents file will be recorded in a
manner similar to the registration of package information in the
current third-party package system. This will allow users to remove
an entire set at a later date, without needing to know what individual
packages came from that set.
Under the current distribution-building system, the Makefile in
/usr/src/etc creates binary install sets from an installed system,
based on the set lists in /usr/src/distrib/sets/lists. In the new
system, a new directory hierarchy, /usr/src/distrib/pkg, will
contain Makefiles and data files relevant to the creation of
system packages and package sets.
The directory /usr/src/distrib/pkg/sets will contain a directory
for each package set, and each of these directories will contain
a directory for each package in that set. The Makefile in
/usr/src/distrib/pkg/sets will recurse into these set directories
to build each set. The individual set Makefiles will recurse into
each package directory to build the individual packages, and will
then create a set file from the constituent packages and from the
contents file, which will be automatically generated from the
package directories.
The package directories will resemble the package directories for
third-party software packages in /usr/pkgsrc, except that they will
probably rely on the files making up the package already being
present in ${DESTDIR}, rather than building them directly. This
assumption is already present in the current distribution package
Makefile code, and is probably reasonable to keep.
4. Modifications to the NetBSD installation process
Once the NetBSD system is available as system packages and package
sets, it will be possible to modify the various installation tools
to use these sets to install the system. It is expected that
installation tools will default to allow users to choose among
package sets at install time, but allow an `advanced mode' in which
packages could be selected and deselected on an individual basis.
This will require that the various package tools (at least pkg_add)
be present on install media to be used with system packages.
Modifications to sysinst and other install tools are beyond the
current scope of this proposal, but will be necessary to take
advantage of the new capabilities provided by this system.
My current plan for implementing system packages and package sets
for NetBSD consists of four steps. All of these steps should be
taken in the CVS source tree (segregated into src/distrib/pkg, of
course), and hopefully will involve other contributors in addition
to myself:
1.) Hammer this proposal into a more detailed specification
I am submitting this proposal now in the hopes that it
will spark discussion which will lead to a refinement
of the planned system package system. Once some sort
of consensus is reached on the relevant mailing lists,
I will begin work in earnest on implementing this.
2.) Create the /usr/src/distrib/pkg hierarchy, and a template
package
The first step in actually implementing this system will
be to create either an actual or mocked-up system package
which can be used as a template for creation of the
remaining system packages.
3.) Create system packages
I expect that this step will involve most of the actual
work in implementing the new system. Packages will have
to be created for each functional group of binaries
currently shipped with NetBSD. A lot of discussion and
design will have to go into the decisions as to how
many packages should make up each set and what files
belong in which packages.
4.) Create Package Sets
Once all system packages exist, it will be necessary to
put together some code to automatically generate set
contents files and to create sets from each directory
of packages in /usr/src/distrib/pkg/sets.
Once these steps are complete, NetBSD will have system packages,
and it will be possible to begin looking at modifying the NetBSD
install process to use them. It is important to note that none
of these changes will require modifying the current installation
set building code in any way, so the use of the current system
can continue unhindered while the new system is being implemented.
