Preface

Preface

I started this paper in 2013, and in 2015 sent it out for review to the
people listed later on. After incorporating comments, I sent it to Rik
Farrow, the editor of the USENIX magazine ;login: to see if he would
publish it. He declined to do so, for reasonably good reasons.

The paper languished, forgotten, until early 2018 when I came across it
and decided to polish it off, put it up on GitHub, and make it
available from my home page in HTML.

If you are interested in language design and evolution in general, and
in Awk in particular, I hope you will enjoy reading this paper. If not,
then why are you bothering looking at it now?

Arnold Robbins
Nof Ayalon, ISRAEL
March, 2018

1 Introduction

At the March 1991 USENIX conference, Henry Spencer presented a paper
entitled AWK As A Major Systems Programming Language. In it, he
described his experiences using the original version of awk to write
two significant “systems” programs—a clone for a reasonable subset of
the nroff formatter[1]^1, and a simple parser generator.

He described what awk did well, as well as what it didn’t, and
presented a list of things that awk would need to acquire in order to
take the position of a reasonable alternative to C for systems
programming tasks on Unix systems.

In particular, awk lies about in the middle of the spectrum between C,
which is “close to the metal,” and the shell, which is quite
high-level. A language at this level that is useful for doing systems
programming is very desirable.

This paper reviews Henry’s wish list, and describes some of the events
that have occurred in the Unix/Linux world since 1991. It presents a
case that gawk—GNU Awk—fills most of the major needs Henry listed way
back in 1991, and then describes the author’s opinion as to why other
languages have successfully filled the systems programming role which
awk did not. It discusses how the current version of gawk may finally
be able to join the ranks of other popular, powerful, scripting
languages in common use today, and ends off with some counter-arguments
and the author’s responses to them.

Acknowledgements

Thanks to Andrew Schorr, Henry Spencer, Nelson H.F. Beebe, and Brian
Kernighan for reviewing an earlier draft of this paper.

2 That Was Then …

In this section we review the state of the Unix world in 1991, as well
as the state of awk, and then list what Henry Spencer saw as missing
for awk.

2.1 The Unix World in 1991

Undoubtedly, many readers of this paper were not using computers in
1991, so this section provides the context in which Henry’s paper was
written. In March of 1991:
* Commercial Unix systems were the norm, with offerings from AT&T,
Digital Equipment Corporation, Hewlett Packard, IBM, Sun
Microsystems, and many others, all vying for market share.
Microsoft Windows existed, but was primarily a layer on top of
MS-DOS and was not taken seriously.
* Very few sites still ran the original Bell Labs or direct-from-UCB
variants of Unix; those did not keep up with the available hardware
and AT&T was itself trying to succeed in the Unix hardware market.
* GNU/Linux did not exist! Some unencumbered BSD variants were
available, but they were still under the cloud of the AT&T/UCB law
suit.[2]^2
* So-called “new” awk was about 2.5 years old. The book by Aho,
Weinberger and Kernighan was published in October of 1987, so most
people knew about new awk, but they just couldn’t get it.
Who could? New awk was available to educational institutions from
the Bell Labs research group, and to those who had Unix source
licenses for System V Releases 3.1, 3.2, and 4. By this time,
source licensees were an extremely rare breed, since the cost for
commercial licenses had skyrocketed, and even for educational
licensees it had increased greatly.[3]^3 If I recall correctly, an
educational license cost around US $1,000, considerably more than
the earlier Unix licenses.
* PERL[4]^4 existed and was starting to gain in popularity. In 1991,
“PERL” most likely meant PERL 3 or a very early version of PERL 4.
The World Wide Web, which was one of the major reasons for PERL’s
growth in popularity, had not yet really taken off.
* Other implementations of new awk were available:
+ + MKS Awk for PC systems (MS-DOS).
+ + GNU Awk was available and relatively stable, but could not
be called “solid.”
The problem with the first of these is that source code was not
available. And the latter came with (to quote Henry) “troublesome
licenses.” (Actually, Henry no longer remembers whether his
statement about “troublesome licenses” referred to the GPL, or to
the Bell Labs source licenses.)
* Michael Brennan’s mawk (also GPL’ed) was not yet available. Version
1.0 was accepted for posting in comp.sources.reviewed on September
30, 1991, half a year after Henry’s paper was published.

2.2 What Awk Lacked In 1991

Here is a summary of what was wrong with the awk picture in 1991. These
are in the same order as presented Henry’s paper. We qualify each issue
in order to later discuss how it has been addressed over time.
1. New awk was not widely available. Most Unix vendors still shipped
only old awk. (Here is where he mentions that “the
independently-available implementations either cost substantial
amounts of money or come with troublesome [sic] licenses.”) His
point then was that for portability, awk programs had to be
restricted to old awk.
This could be considered a quality of implementation issue,
although it’s really a “lack of available implementation” issue.
2. There is no way to tell awk to start matching all its patterns over
again against the existing $0. This is a language design issue.
3. There is no array assignment. (Language design issue.)
4. Getting an error message out to standard error is difficult.
(Implementation issue.)
5. There is no precise language specification for awk. This leads to
gratuitous portability problems. This too is thus a quality of
implementation issue, in that without a specification, it’s
difficult to produce uniform, high quality implementations.
6. The existing widely available implementation is slow; a much faster
implementation is needed and the best thing of all would be an
optimizing compiler. (Implementation issue.)
7. There is no awk-level debugger. (Support tool or quality of
implementation issue.)
8. There is no awk-level profiler. (Support tool or quality of
implementation issue.)

In private email, Henry added the following items, saying “there are a
couple more things I’d add now, in hindsight.” These are direct quotes:
9. [I can’t believe I didn’t discuss this in the paper, because I was
certainly aware of it then!] Lack of any convenient mechanism for
adding libraries. When awk is being invoked from a shell file, the
shell file can do substitutions or use multiple -f options, but
those are mechanisms outside the language, and not very convenient
ones. What’s really wanted is something like you get in Python
etc., where one little statement up near the top says “arrange for
this program to have the xyz library available when it runs.”
10. I think it was Rob Pike who later said (roughly): “It says
something bad about Awk that in a language with integrated regular
expressions, you end up using substr() so often.” My paper did
allude to the difficulty of finding out where something matched in
old-awk programs, but even in new awk, what you get is a number
that you then have to feed to substr(). The language could really
use some more convenient way of dissecting a string using regexp
matching. [Caveat: I have not looked lately at Gawk to see if it
has one.]

The first of these is somewhere between a language design and a
language implementation issue. The latter is a language design issue.

3 … And This Is Now

Fast forward to 2018. Where do things stand?

3.1 What Awk Has Today

The state of the awk world is much better now. In the same order:
1. New awk is the standard version of awk today on GNU/Linux, BSD, and
commercial Unix systems. The one notable exception is Solaris,
where /usr/bin/awk is still the old one; on all other systems,
plain awk is some version of new awk.
2. There remains no way to tell awk to start matching all its patterns
over again against the existing $0. Furthermore, this is a feature
that has not been called for by the awk community, except in
Henry’s paper. (We do acknowledge that this might be a useful
feature.)
3. There continues to be no array assignment. However, this function
in gawk, which has arrays of arrays, can do the trick nicely. It is
also efficient, since gawk uses reference counted strings
internally:
function copy_array(dest, source, i, count)
{
delete dest

for (i in source) {
if (typeof(source[i]) == "array")
count += copy_array(dest[i], source[i])
else {
dest[i] = source[i]
count++
}
}

return count
}

4. Getting error messages out is easier. All modern systems have a
/dev/stderr special file to which error messages may be sent
directly.
5. Perhaps most important of all, with the [5]POSIX standard, there is
a formal standard specification for awk. As with all formal
standards, it isn’t perfect. But it provides an excellent starting
point, as well as chapter and verse to cite when explaining the
behavior of a standards-compliant version of awk.
6. There are a number of freely available implementations, with
different licenses, such that everyone ought to be able to find a
suitable one:
+ Brian Kernighan’s awk is the direct lineal descendant of Unix
awk. He calls it the “One True Awk” (sic). It is available
from [6]his home page, in several archive formats:
Shell archive:
[7]http://www.cs.princeton.edu/~bwk/btl.mirror/awk.sha
r
Compressed tar file:
[8]http://www.cs.princeton.edu/~bwk/btl.mirror/awk.tar
.gz
Zip file:
[9]http://www.cs.princeton.edu/~bwk/btl.mirror/awk.zip
Git Hub:
git clone git://github.com/onetrueawk/awk bwkawk
+ GNU Awk, gawk, is available from the Free Software Foundation.
You may use either ftp or an HTTP downloader:
[10]http://ftp.gnu.org/gnu/gawk/gawk-4.2.1.tar.gz is the
current version. There may be a newer one.
+ Michael Brennan’s awk, known as mawk. In 2009, Thomas Dickey
took on mawk maintenance. Basic information is available on
[11]the project’s web page. The download URL is
[12]http://invisible-island.net/datafiles/release/mawk.tar.gz.
In 2017 Michael published a beta of mawk 2.0. It’s available
from the project’s [13]GitHub page.
+ MKS Awk was used for Solaris’s /usr/xpg4/bin/awk, which is
their standards-compliant version of new awk. For a while it
was available as part of Open Solaris, but is no longer so.
Some years ago, we were able to make this version compile and
run on GNU/Linux after just a few hours work.
Although Open Solaris is now history, the [14]Illumos project
does make the MKS Awk available. You can view the files one at
a time from
[15]https://github.com/joyent/illumos-joyent/blob/master/usr/s
rc/cmd/awk_xpg4.
+ Other, more esoteric versions as well. See the [16]Wikipedia
article, and also the [17]gawk documentation.

3.2 And What GNU Awk Has Today

The more difficult of the quality of implementation issues are
addressed by gawk. In particular:
7. Beginning with version 4.0 in 2011, gawk provides an awk-level
debugger: dgawk, which is modeled after GDB. This is a full
debugger, with breakpoints, watchpoints, single statement stepping
and expression evaluation capabilities.
8. gawk has provided an awk-level statement profiler for many years
(pgawk). Although there is no direct correlation with CPU time
used, the statement level profiler remains a powerful tool for
understanding program behavior.
9. Since version 4.0, gawk has had an ‘@include’ facility whereby gawk
goes and finds the named awk source progrm. For much longer it has
searched for files specified with -f along the path named by the
AWKPATH environment variable. The ‘@include’ mechanism also uses
AWKPATH.
10. In terms of getting at the pieces of text matched by a regular
expression, gawk provides an optional third argument to the match()
function. This argument is an array which gawk fills in with both
the matched text for the full regexp and subexpressions, and index
and length information for use with substr(). gawk also provides
the gensub() general substitution function, an enhanced version of
the split() function, and the patsplit() function for specifying
contents instead of separators using a regexp.

With the 4.1 release, all three versions (gawk, pgawk, and dgawk) are
merged into a single executable, considerably reducing the required
installation “footprint.”

While gawk has almost always been faster than Brian Kernighan’s awk,
recent performance improvements bring it closer to mawk’s performance
level (a byte-code based execution engine and internal improvements in
array indexing).

And gawk clearly has the most features of any version, many of which
considerably increase the power of the language.

3.3 So Where Does Awk Stand?

Despite all of the above, gawk is not as popular as other scripting
languages. Since 1991, we can point to four major scripting languages
which have enjoyed, or currently enjoy, differing levels of popularity:
PERL, tcl/tk, Python, and Ruby. We think it is fair to say that Python
and Ruby are the most popular scripting languages in the second decade
of the 21st century.

Is awk, as we’ve described it up to this point, now ready to compete
with the other languages? Not quite yet.

4 Key Reasons Why Other Languages Have Gained Popularity

In retrospect, it seems clear (at least to us!) that there are two
major reasons that all of the previously mentioned languages have
enjoyed significant popularity. The first is their extensibility. The
second is namespace management.

One certainly cannot attribute their popularity to improved syntax. In
the opinion of many, PERL and Ruby both suffer from terrible syntax.
Tcl’s syntax is readable but nothing special. Python’s syntax is
elegant, although slightly unusual. The point here is that they all
differ greatly in syntax, and none really offers the clean
pattern–action paradigm that is awk’s trademark, yet they are all
popular languages.

If not syntax, then what? We believe that their popularity stems from
the fact that all of these languages are easily extensible. This is
true with both “modules” in the scripting language, and more
importantly, with access to C level facilities via dynamic library
loading.

Furthermore, these languages allow you to group related functions and
variables into packages or modules: they let you manage the namespace.

awk, on the other hand, has always been closed. An awk program cannot
even change its working directory, much less open a connection to an
SQL database or a socket to a server on the Internet somewhere
(although gawk can do the latter).

If one examines the number of extensions available for PERL on CPAN, or
for Python such as PyQt or the Python tk bindings, it becomes clear
that extensibility is the real key to power (and from there to
popularity).

Furthermore, in awk, all global variables and functions share a single
namespace. This prevents many good software development practices based
on the principle of information hiding.

To summarize: A reasonable language definition, efficient
implementations, debuggers and profilers are necessary but not
sufficient for true power. The final ingredients are extensibility and
namespaces.

5 Filling The Extensibility Gap

With version 4.1, gawk (finally) provides a defined C API for extending
the core language.

5.1 API Overview

The API makes it possible to write functions in C or C++ that are
callable from an awk program as if the function were written in awk.
The most straightforward way to think of these functions is as
user-defined functions that happen to be implemented in a different
language.

The API provides the following facilities:
* Structures that map awk string, numeric, and undefined values into
C types that can be worked with.
* Management of function parameters, including the ability to convert
a parameter whose original type is undefined, into an array. That
is, there is full call-by-reference for arrays. Scalars are passed
by value, of course.
* Access to the symbol table. Extension functions can read all awk
variables, and create and update new variables. As an initial,
relatively arbitrary design decision, extensions cannot update
special variables such as NR or NF, with the single exception of
PROCINFO.
* Full array management, including the ability to create arrays, and
arrays of arrays, and the ability to add and delete elements from
an array. It is also possible to “flatten” an array into a data
structure that makes it simple for C code to loop over all the
elements of an array.
* The ability to run a procedure when gawk exits. This is
conceptually the same as the C atexit() function.
* Hooks into the built-in I/O redirection mechanisms in gawk. In
particular, there are separate facilities for input redirections
with getline and ‘<’, output redirections with print or printf and
‘>’ or ‘>>’, and two-way pipelines with gawk’s ‘|&’ operator.

5.2 Discussion

Considerable thought went into the design of the API. The gawk
documentation provides a [18]full description of the API itself, with
examples (over 50 pages worth!), as well as [19]some discussion of the
goals and design decisions behind the API (in an appendix). The
development was done over the course of about a year and a half,
together with the developers of xgawk, a fork of gawk that added
features that made using extensions easier, and included an extension
for processing XML files in a way that fit naturally with the
pattern–action paradigm. While it may not be perfect, the gawk
developers feel that it is a good start.

FIXME: Henry Spencer suggests adding more info on the API and on the
design decisions. I think this paper is long enough, and the full doc
is quite big. It’d be hard to pull API doc into this paper in a
reasonable fashion, although it would be possible to review some of the
design decisions. Comments?

The major xgawk additions to the C code base have been merged into
gawk, and the extensions from that project have been rewritten to use
the new API. As a result, the xgawk project developers renamed their
project gawkextlib, and the project now provides only extensions.[20]^5

It is notable that functions written in awk can do a number of things
that extension functions cannot, such as modify any variables, do I/O,
call awk built-in functions, and call other user-defined functions.

While it would certainly be possible to provide APIs for all of these
features for extension functions, this seemed to be overkill. Instead,
the gawk developers took the view that extension functions should
provide access to external facilities, and provide communication to the
awk level via function parameters and/or global variables, including
associative arrays, which are the only real data structure.

Consider a simple example. The standard du program can recursively walk
one or more arbitrary file hierarchies, call stat() to retrieve file
information, and then sum up the blocks used. In the process, du must
track hard links, so that no file is accounted for or reported more
than once.

The ‘filefuncs’ extension shipped with gawk provides a stat() function
that takes a pathname and fills in an associative array with the
information retrieved from stat(). The array elements have names like
"size", "mtime" and so on, with corresponding appropriate values.
(Compare this to PERL’s stat() function that returns a linearly-indexed
array!)

The fts() function in the ‘filefuncs’ extension builds on stat() to
create a multidimensional array of arrays that describes the requested
file hierarchies, with each element being an array filled in by stat().
Directories are arrays containing elements for each directory entry,
with an element named "." for the array itself.

Given that fts() does the heavy lifting, du can be written quite
nicely, and quite portably[21]^6, in awk. See [22]du in awk, for the
code, which weighs in at under 250 lines. Much of this is comments and
argument parsing.

5.3 Future Work

The extension facility is relatively new, and undoubtedly has
introduced new “dark corners” into gawk. These remain to be uncovered
and any new bugs need to be shaken out and removed.

Some issues are known and may not be resolvable. For example, 64-bit
integer values such as the timestamps in stat() data on modern systems
don’t fit into awk’s 64-bit double-precision numbers which only have 53
bits of significand. This is also a problem for the bit-manipulation
functions.

With respect to namespaces, in 2017 I (finally) figured out how
namespaces in awk ought to work to provide the needed functionality
while retaining backwards compatibility. The code is currently in the
feature/namespaces branch of gawk’s Git repository. It will eventually
be merged into the master branch for release as part of gawk 5.0.

FIXME: More info needed here.

6 Counterpoints

Brian Kernighan raised several counterpoints in response to an earlier
draft of the paper. They are worth addressing (or at least trying to):

I’m not 100% convinced by your basic premise, that the lack of an
extension mechanism is the main / a big reason why Awk isn’t used
for the kinds of system programming tasks that Perl, Python, etc.,
are. It’s absolutely a factor—without such a mechanism, there’s just
no way to do a lot of important computations. But how does that
trade off against just having built-in mechanisms for the core
system programming facilities (as Perl does) or a handful of core
libraries like sys, os, regex, etc., for Python?

I think that Perl’s original inclusion of most of the Unix system calls
was, from a language design standpoint, ultimately a mistake. At the
time it was first done, there was no other choice: dynamic loading of
libraries didn’t exist on Unix systems in the early and mid-1980s (nor
did shared libraries, for that matter). But having all those built-in
functions bloats the language, making it harder to learn, document, and
maintain, and I definitely did not wish to go down that path for gawk.

With respect to Python, the question is: how are those libraries
implemented? Are they built-in to the interpreter and separated from
the “core” language simply by the language design? Or are they
dynamically loaded modules?

If the latter, that sounds like an argument for the case of having
extensions, not against it. And indeed, this merely emphasizes the
point made at the end of the previous section, which is that to make an
extension facility really scalable, you also need some sort of
namespace / module capability.

Thus, Brian is correct: an extension facility is needed, but the last
part of the puzzle would be a module facility in the language. I think
that I have solved this, and invite the curious reader to checkout the
branch named earlier and provide feedback.

I’m also not convinced that Awk is the right language for writing
things that need extensions. It was originally designed for
1-liners, and a lot of its constructs don’t scale up to bigger
programs. The notation for function locals is appalling (all my
fault too, which makes it worse). There’s little chance to recover
from random spelling mistakes and typos; the use of mere adjacency
for concatenation looks ever more like a bad idea.

This is hard to argue with. Nonetheless, gawk’s --lint option may be of
help here, as well as the --dump-variables option which produces a list
of all variables used in the program.

Awk is fine for its original purpose, but I find myself writing
Python for anything that’s going to be bigger than say 10-20 lines
unless the lines are basically just longer pattern-action sequences.
(That notation is a win, of course, which you point out.)

Since my Python experience is minimal, I have little to say here; it
might be that if I were more familiar with Python, I would start using
it for small scripts instead of awk.

On the other hand, with discipline, it’s possible to write fairly
good-sized, understandable and maintainable awk programs; in my
experience awk does scale up well beyond the one-liner range.

Not to mention that Brian published a whole book of awk programs larger
than one line. :-) (See the Resources section.)

Some of my own, good-sized awk programs are available from GitHub:
The TexiWeb Jr. literate programming system
See [23]https://github.com/arnoldrobbins/texiwebjr. The suite
has two programs that total over 1,300 lines of awk. (They share
some code.)
Prepinfo
See [24]https://github.com/arnoldrobbins/prepinfo. This script
processes Texinfo files, updating menus as needed. This version
is rewritten in TexiWeb Jr.; it’s about 350 lines of awk.
Sortmail
See [25]https://github.com/arnoldrobbins/sortmail. This script
sorts a Unix mbox format mailbox by thread. I use it daily. It’s
also written in TexiWeb Jr. and is about 330 lines of awk.

Brian continues:

The du example is awfully big, though it does show off some of the
language features. Could you get the same mileage with something
quite a bit shorter?

My definition of “small” and “big” has changed over time. 250 lines may
be big for a script, but the du.awk program is much smaller than a full
implementation in C: GNU du is over 1,100 lines of C, plus all the
libraries it relies upon in the GNU Coreutils.

With respect to shorter examples, nothing springs to mind immediately.
However, gawk comes with several useful extensions that are worth
exploring, much more than we’ve covered here.

For example, the readdir extension in the gawk distribution causes gawk
to read directories and return one record per directory entry in an
easy-to-parse format:
$ gawk -lreaddir '{ print }' .
-| 2109292/mail.mbx/f
-| 2109295/awk-sys-prog.texi/f
-| 2100007/./d
-| 2100056/texinfo.tex/f
-| 2100055/cleanit/f
-| 2109282/awk-sys-prog.pdf/f
-| 2100009/du.awk/f
-| 2100010/.git/d
-| 2098025/../d
-| 2109294/ChangeLog/f

How cool is that?!? :-)

Also, the gawkextlib project provides some very interesting extensions.
Of particular interest are the XML and JSON extensions, but there are a
number of others, and it’s worth checking out.

In short, it’s too early to really tell. This is the beginning of an
experiment. I hope it will be a fun journey for me, the other gawk
maintainers, and the larger community of awk users.

7 Conclusion

It has taken much longer than any awk fan would like, but finally, GNU
Awk fills in almost all the gaps listed by Henry Spencer for awk to be
really useful as a systems programming language.

In addition, experience from other popular languages has shown that
extensibility and namespaces are the keys to true power, usability, and
popularity.

With the release of gawk 4.1, we feel that gawk (and thus the Awk
language) are now almost on par with the basic capabilities of other
popular languages. With gawk 5.0, we hope to truly reach par.

Is it too late in the game? If enough people start to write extensions
for gawk, then perhaps awk will return to the scripting language
limelight. If not, then the gawk developers will have wasted a lot of
time and effort. (We hope not!) Time will tell.

For now though, we hope that this paper will have piqued your
curiosity, and that you will take the time to give gawk a fresh look.

Appendix A Resources

1. The AWK Programming Language Paperback, Alfred V. Aho, Brian W.
Kernighan, and Peter J. Weinberger. Addison-Wesley, 1988. ISBN-13:
978-0201079814, ISBN-10: 020107981X.
2. Effective awk Programming, fourth edition. Arnold Robbins. O’Reilly
Media, 2015. ISBN-13: 978-1491904619, ISBN-10: 1491904615.
3. Online version of the gawk documentation:
[26]http://www.gnu.org/software/gawk/manual/.
4. The gawkextlib project:
[27]http://sourceforge.net/projects/gawkextlib/.

Appendix B Awk Code For du

Here is the du program, written in Awk. Besides demonstrating the power
of the stat() and fts() extensions and gawk’s multidimensional arrays,
it also shows the switch statement and the built-in bit manipulation
functions and(), or(), and compl().

The output is not identical to GNU du’s, since filenames are not
sorted. However, gawk’s built-in sorting facilities should make sorting
the output straightforward; we leave that as the traditional “exercise
for the reader.”
#! /usr/local/bin/gawk -f

# du.awk --- write POSIX du utility in awk.
# See http://pubs.opengroup.org/onlinepubs/9699919799/utilities/du.html
#
# Most of the heavy lifting is done by the fts() function in the "filefuncs"
# extension.
#
# We think this conforms to POSIX, except for the default block size, which
# is set to 1024. Following GNU standards, set POSIXLY_CORRECT in the
# environment to force 512-byte blocks.
#
# Arnold Robbins
# [email protected]

@include "getopt"
@load "filefuncs"

BEGIN {
FALSE = 0
TRUE = 1

BLOCK_SIZE = 1024 # Sane default for the past 30 years
if ("POSIXLY_CORRECT" in ENVIRON)
BLOCK_SIZE = 512 # POSIX default

compute_scale()

fts_flags = FTS_PHYSICAL
sum_only = FALSE
all_files = FALSE

while ((c = getopt(ARGC, ARGV, "aHkLsx")) != -1) {
switch (c) {
case "a":
# report size of all files
all_files = TRUE;
break
case "H":
# follow symbolic links named on the command line
fts_flags = or(fts_flags, FTS_COMFOLLOW)
break
case "k":
BLOCK_SIZE = 1024 # 1K block size
break
case "L":
# follow all symbolic links

# fts_flags &= ~FTS_PHYSICAL
fts_flags = and(fts_flags, compl(FTS_PHYSICAL))

# fts_flags |= FTS_LOGICAL
fts_flags = or(fts_flags, FTS_LOGICAL)
break
case "s":
# do sums only
sum_only = TRUE
break
case "x":
# don't cross filesystems
fts_flags = or(fts_flags, FTS_XDEV)
break
case "?":
default:
usage()
break
}
}

# if both -a and -s
if (all_files && sum_only)
usage()

for (i = 0; i < Optind; i++)
delete ARGV[i]

if (Optind >= ARGC) {
delete ARGV # clear all, just to be safe
ARGV[1] = "." # default to current directory
}

fts(ARGV, fts_flags, filedata) # all the magic happens here

# now walk the trees
if (sum_only)
sum_walk(filedata)
else if (all_files)
all_walk(filedata)
else
top_walk(filedata)
}

# usage --- print a message and die

function usage()
{
print "usage: du [-a|-s] [-kx] [-H|-L] [file] ..." > "/dev/stderr"
exit 1
}

# compute_scale --- compute the scale factor for block size calculations

function compute_scale( stat_info, blocksize)
{
stat(".", stat_info)

if (! ("devbsize" in stat_info)) {
printf("du.awk: you must be using filefuncs extension from gawk 4.1.1 or
later\n") > "/dev/stderr"
exit 1
}

# Use "devbsize", which is the units for the count of blocks
# in "blocks".
blocksize = stat_info["devbsize"]
if (blocksize > BLOCK_SIZE)
SCALE = blocksize / BLOCK_SIZE
else # I can't really imagine this would be true
SCALE = BLOCK_SIZE / blocksize
}

# islinked --- return true if a file has been seen already

function islinked(stat_info, device, inode, ret)
{
device = stat_info["dev"]
inode = stat_info["ino"]

ret = ((device, inode) in Files_seen)

return ret
}

# file_blocks --- return number of blocks if a file has not been seen yet

function file_blocks(stat_info, device, inode)
{
if (islinked(stat_info))
return 0

device = stat_info["dev"]
inode = stat_info["ino"]

Files_seen[device, inode]++

return block_count(stat_info) # delegate actual counting
}

# block_count --- return number of blocks from a stat() result array

function block_count(stat_info, result)
{
if ("blocks" in stat_info)
result = int(stat_info["blocks"] / SCALE)
else
# otherwise round up from size
result = int((stat_info["size"] + (BLOCK_SIZE - 1)) / BLOCK_SIZE)

return result
}

# sum_dir --- data on a single directory

function sum_dir(directory, do_print, i, sum, count)
{
for (i in directory) {
if ("." in directory[i]) { # directory
count = sum_dir(directory[i], do_print)
count += file_blocks(directory[i]["."])
if (do_print)
printf("%d\t%s\n", count, directory[i]["."]["path"])
} else { # regular file
count = file_blocks(directory[i]["stat"])
}
sum += count
}

return sum
}

# simple_walk --- summarize directories --- print info per parameter

function simple_walk(filedata, do_print, i, sum, path)
{
for (i in filedata) {
if ("." in filedata[i]) { # directory
sum = sum_dir(filedata[i], do_print)
path = filedata[i]["."]["path"]
} else { # regular file
sum = file_blocks(filedata[i]["stat"])
path = filedata[i]["path"]
}
printf("%d\t%s\n", sum, path)
}
}

# sum_walk --- summarize directories --- print info only for the top set of dire
ctories

function sum_walk(filedata)
{
simple_walk(filedata, FALSE)
}

# top_walk --- data on the main arguments only

function top_walk(filedata)
{
simple_walk(filedata, TRUE)
}

# all_walk --- data on every file

function all_walk(filedata, i, sum, count)
{
for (i in filedata) {
if ("." in filedata[i]) { # directory
count = all_walk(filedata[i])
sum += count
printf("%s\t%s\n", count, filedata[i]["."]["path"])
} else { # regular file
if (! islinked(filedata[i]["stat"])) {
count = file_blocks(filedata[i]["stat"])
sum += count
if (i != ".")
printf("%d\t%s\n", count, filedata[i]["path"])
}
}
}
return sum
}
__________________________________________________________________

References

1. file:///tmp/lynxXXXXQrChxN/L713986-2883TMP.html#FOOT1
2. file:///tmp/lynxXXXXQrChxN/L713986-2883TMP.html#FOOT2
3. file:///tmp/lynxXXXXQrChxN/L713986-2883TMP.html#FOOT3
4. file:///tmp/lynxXXXXQrChxN/L713986-2883TMP.html#FOOT4
5. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html
6. http://www.cs.princeton.edu/~bwk
7. http://www.cs.princeton.edu/~bwk/btl.mirror/awk.shar
8. http://www.cs.princeton.edu/~bwk/btl.mirror/awk.tar.gz
9. http://www.cs.princeton.edu/~bwk/btl.mirror/awk.zip
10. http://ftp.gnu.org/gnu/gawk/gawk-4.2.1.tar.gz
11. http://www.invisible-island.net/mawk
12. http://invisible-island.net/datafiles/release/mawk.tar.gz
13. https://github.com/mikebrennan000/mawk-2
14. http://wiki.illumos.org/display/illumos/illumos+Home
15. https://github.com/joyent/illumos-joyent/blob/master/usr/src/cmd/awk_xpg4
16. http://en.wikipedia.org/wiki/Awk_language#Versions_and_implementations
17. http://www.gnu.org/software/gawk/manual/html_node/Other-Versions.html#Other-Versions
18. http://www.gnu.org/software/gawk/manual/html_node/Dynamic-Extensions.html#Dynamic-Extensions
19. http://www.gnu.org/software/gawk/manual/html_node/Extension-Design.html#Extension-Design
20. file:///tmp/lynxXXXXQrChxN/L713986-2883TMP.html#FOOT5
21. file:///tmp/lynxXXXXQrChxN/L713986-2883TMP.html#FOOT6
22. file:///tmp/lynxXXXXQrChxN/L713986-2883TMP.html#du-in-awk
23. https://github.com/arnoldrobbins/texiwebjr
24. https://github.com/arnoldrobbins/prepinfo
25. https://github.com/arnoldrobbins/sortmail
26. http://www.gnu.org/software/gawk/manual/
27. http://sourceforge.net/projects/gawkextlib/