Linux Ext2fs Undeletion mini-HOWTO

Linux Ext2fs Undeletion mini-HOWTO
Aaron Crane, <[email protected]>
v1.2, 4 August 1997

Picture this. You've spent the last three days with no sleep, no
food, not even a shower. Your hacking compulsion has at last paid
off: you've finished that program that will bring you world-wide fame
and recognition. All that you still need to do is tar it up and put
it on Sunsite. Oh, and delete all those Emacs backup files. So you
say rm * ~. And too late, you notice the extra space in that command.
You've just deleted your magnum opus! But help is at hand. This doc�
ument presents a discussion of how to retrieve deleted files from a
Second Extended File System. Just maybe, you'll be able to release
that program after all...

1. Introduction

This mini-Howto attempts to provide hints on how to retrieve deleted
files from an ext2 filesystem. It also contains a limited amount of
discussion of how to avoid deleting files in the first place.

I intend it to be useful certainly for people who have just had, shall
we say, a little accident with rm; however, I also hope that people
read it anyway. You never know: one day, some of the information in
here could save your bacon.

The text assumes a little background knowledge about UNIX filesystems
in general; however, I hope that it will be accessible to most Linux
users. If you are an outright beginner, I'm afraid that undeleting
files under Linux does require a certain amount of technical knowledge
and persistence, at least for the time being.

You will be unable to recover deleted files from an ext2 filesystem
without at least read access to the raw device on which the file was
stored. In general, this means that you must be root. You also need
debugfs from the e2fsprogs package. This should have been installed
by your distribution.

Why have I written this? It stems largely from my own experiences
with a particularly foolish and disastrous rm -r command as root. I
deleted about 97 JPEG files which I needed and could almost certainly
not recover from other sources. Using some helpful tips (see section
``Credits and Bibliography'') and a great deal of persistence, I
recovered 91 files undamaged. I managed to retrieve at least parts of
five of the rest (enough to see what the picture was in each case).
Only one was undisplayable, and even for this one, I am fairly sure
that no more than 1024 bytes were lost (though unfortunately from the
beginning of the file; given that I know nothing about the JFIF file
format I had done as much as I could).

I shall discuss further below what sort of recovery rate you can
expect for deleted files.

1.1. Revision history

The various publicly-released revisions of this document (and their
publication dates) are as follows:

� v1.0 on 18 January 1997

� v1.1 on 23 July 1997 (see section ``Changes in v1.1'')

� v1.2 on 4 August 1997 (see section ``Changes in v1.2'')

1.1.1. Changes in version 1.1

What changes have been made in this version? First of all, the thinko
in the example of file recovery has been fixed. Thankyou to all those
who wrote to point out my mistaek; I hope I've learned to be more
careful when making up program interaction.

Secondly, the discussion of UNIX filesystem layout has been rewritten
to be, I hope, more understandable. I wasn't entirely happy with it
in the first place, and some people's comments indicated that it
wasn't clear.

Thirdly, the vast uuencoded gzipped tarball of fsgrab in the middle of
the file has been removed. The program is now available on my website
<http://pobox.com/~aaronc/tech/fsgrab-1.0.tar.gz> and it should soon
make its way onto Sunsite
<http://sunsite.unc.edu/pub/Linux/utils/disk-management/> (and
mirrors).

Fourthly, the document has been translated into the Linux
Documentation Project SGML Tools content markup language. This markup
language can be easily converted to any of a number of other markup
languages (including HTML and LaTeX) for convenient display and
printing. One benefit of this is that beautiful typography in paper
editions is a much more achievable goal; another is that the document
has cross-references and hyperlinks when viewed on the Web.

1.1.2. Changes in v1.2

This revision is very much an incremental change. It's here mainly to
include changes suggested by readers, one of which is particularly
important.

The first change was suggested by Egil Kvaleberg <[email protected]>,
who pointed out the dump command in debugfs. Thanks again, Egil.

The second change is to mention the use of chattr for avoiding
deleting important files. Thanks to Herman Suijs <[email protected]>
for mentioning this one.

The abstract has been revised. URLs have been added for organisations
and software. Various other minor changes have been made (including
fixing typos and so on).

1.2. Canonical locations of this document

The latest public release of this document should always be available
in plain text format on the Linux Documentation Project site
<http://sunsite.unc.edu/LDP/> (and mirrors).

The latest release is also kept on my website
<http://pobox.com/~aaronc/> in several formats:

� SGML source <http://pobox.com/~aaronc/tech/e2-undel/howto.sgml>.
This is the source as I have written it, using the SGML Tools
package.

� HTML <http://pobox.com/~aaronc/tech/e2-undel/html/>. This is HTML,
automatically generated from the SGML source.

� Plain text <http://pobox.com/~aaronc/tech/e2-undel/howto.txt>.
This is plain text, which is also automatically generated from the
SGML source. Note that this file is identical to the one on
Sunsite, so if you want the plain text, you are recommended to get
it from your favourite LDP mirror (as it will probably be much
faster).

2. How not to delete files

It is vital to remember that Linux is unlike MS-DOS when it comes to
undeletion. For MS-DOS (and its bastard progeny Windows 95), it is
generally fairly straightforward to undelete a file - the `operating
system' (I use the term loosely) even comes with a utility which
automates much of the process. For Linux, this is not the case.

So. Rule number one (the prime directive, if you will) is:

KEEP BACKUPS

no matter what. I know, I'm a fine one to talk. I shall merely plead
impoverishment (being a student must have some perks) and exhort all
right-thinking Linux users to go out and buy a useful backup device,
work out a decent backup schedule, and to stick to it. For more
information on this, read Frisch (1995) (see section ``Bibliography
and Credits'').

In the absence of backups, what then? (Or even in the presence of
backups: belt and braces is no bad policy where important data is
concerned.)

Try to set the permissions for important files to 440 (or less):
denying yourself write access to them means that rm requires an
explicit confirmation before deleting. (I find, however, that if I'm
recursively deleting a directory with rm -r, I'll interrupt the
program on the first or second confirmation request and reissue the
command as rm -rf.)

A good trick for selected files is to create a hard link to them in a
hidden directory. I heard a story once about a sysadmin who
repeatedly deleted /etc/passwd by accident (thereby half-destroying
the system). One of the fixes for this was to do something like the
following (as root):

# mkdir /.backup
# ln /etc/passwd /.backup

It requires quite some effort to delete the file contents completely:
if you say

# rm /etc/passwd

then

# ln /.backup/passwd /etc

will retrieve it. Of course, this does not help in the event that you
overwrite the file, so keep backups anyway.

On an ext2 filesystem, it is possible to use ext2 attributes to
protect things. These attributes are manipulated with the chattr
command. There is an `append-only' attribute: a file with this
attribute may be appended to, but may not be deleted, and the existing
contents of the file may not be overwritten. If a directory has this
attribute, any files or directories within it may be modified as
normal, but no files may be deleted. The `append-only' attribute is
set with

$ chattr +a FILE...

There is also an `immutable' attribute, which can only be set or
cleared by root. A file or directory with this attribute may not be
modified, deleted, renamed, or (hard) linked. It may be set as
follows:

# chattr +i FILE...

The ext2fs also provides the `undeletable' attribute (+u in chattr).
The intention is that if a file with that attribute is deleted,
instead of actually being reused, it is merely moved to a `safe
location' for deletion at a later date. Unfortunately this feature
has not yet been implemented in mainstream kernels. However, various
kernel patches exist which provide the ability to do reversible
deletion; see <http://www.linuxhq.com/> if you're interested in
patching this facility into your kernel. The most recent patch I know
of is by Rogier Wolff <[email protected]>, Darren J Moffat
<[email protected]> and Kurt Huwig <[email protected]>. I would
point out though that while this patch implements the feature, it is
not an `undeletion solution' at the moment. Undeletable files are
merely moved into another directory; there should be a daemon to
periodically clean up that directory.

Some people advocate making rm a shell alias or function for rm -i
(which asks for confirmation on every file you delete). Indeed,
recent versions of the Red Hat distribution <http://www.redhat.com/>
do this by default for all users, including root. Personally, I
cannot stand software which won't run unattended, so I don't do that.
There is also the problem that sooner or later, you'll be running in
single-user mode, or using a different shell, or even a different
machine, where your rm function doesn't exist. If you expect to be
asked for confirmation, it is easy to forget where you are and to
specify too many files for deletion. Likewise, the various scripts
and programs that replace rm are, IMHO, very dangerous.

A slightly better solution is to start using a package which handles
`recyclable' deletion by providing a command not named rm. For
details on these, see Peek, et al (1993) (see section ``Bibliography
and Credits'').

3. What recovery rate can I expect?

That depends. Among the problems with recovering files on a high-
quality, multi-tasking, multi-user operating system like Linux is that
you never know when someone wants to write to the disk. So when the
operating system is told to delete a file, it assumes that the blocks
used by that file are fair game when it wants to allocate space for a
new file. (This is a specific example of a general principle for
Linux: the kernel and the associated tools assume that the users
aren't idiots.) In general, the more usage your machine gets, the
less likely you are to be able to recover files successfully.

Also, disk fragmentation can affect the ease of recovering files. If
the partition containing the deleted files is very fragmented, you are
unlikely to be able to read a whole file.

If your machine, like mine, is effectively a single-user workstation
(mine doesn't even have a net connection yet; maybe next year), and
you weren't doing anything disk-intensive at the fatal moment of
deleting those files, I would expect a recovery rate in the same ball-
park as detailed above. I retrieved nearly 94% of the files (and
these were binary files, please note) undamaged. If you get 80% or
better, you can feel pretty pleased with yourself, I should think.

4. So, how do I undelete a file?

The procedure principally involves finding the data on the raw
partition device and making it visible again to the operating system.
There are basically two ways of doing this: one is to modify the
existing filesystem such that the deleted inodes have their `deleted'
flag removed, and hope that the data just magically falls back into
place. The other method, which is safer but slower, is to work out
where the data lies in the partition and write it out into a new file.

There are some steps you need to take before beginning to attempt your
data recovery; see sections ``Unmounting the filesystem'', ``Preparing
to change inodes directly'' and ``Preparing to write data elsewhere''
for details. To find out how to actually retrieve your files, see
sections ``Finding the deleted inodes'', ``Obtaining the details of
the inodes'', ``Recovering data blocks'' and ``Modifying inodes
directly''.

5. Unmounting the filesystem

Regardless of which method you choose, the first step is to unmount
the filesystem containing the deleted files. I strongly discourage
any urges you may have to mess around on a mounted filesystem. This
step should be performed as soon as possible after you realise that
the files have been deleted.

The simplest method is as follows: assuming the deleted files were in
the /usr partition, say:

# umount /usr

You may, however, want to keep some things in /usr available. So
remount it read-only:

# mount -o ro,remount /usr

If the deleted files were on the root partition, you'll need to add a
-n option to prevent mount from trying to write to /etc/mtab:

# mount -n -o ro,remount /

Regardless of all this, it is possible that there will be another
process using that filesystem (which will cause the unmount to fail
with an error such as `Resource busy'). There is a program which will
send a signal to any process using a given file or mount point: fuser.
Try this for the /usr partition:

# fuser -v -m /usr

This lists the processes involved. Assuming none of them are vital,
you can say

# fuser -k -v -m /usr

to send each process a SIGKILL (which is guaranteed to kill it), or
for example,

# fuser -k -TERM -v -m /usr

to give each one a SIGTERM (which will normally make the process exit
cleanly).

6. Preparing to change inodes directly

My advice? Don't do it this way. I really don't think it's wise to
play with a filesystem at a low enough level for this to work. There
are also has problems in that you can only reliably recover the first
12 blocks of each file. So if you have any long files to recover,
you'll have to use the other method anyway. (Although see section
``Will this get easier in future?'' for additional information.)

If you feel you must do it this way, my advice is to copy the raw
partition data to an image on a different partition, and then mount
this using loopback:

# cp /dev/hda5 /root/working
# mount -t ext2 -o loop /root/working /mnt

This does, however, require a recent version of mount. (Although you
should get version 2.6 or newer anyway, as all earlier versions have a
major security bug which allows peons to get root access. The major
distributions, that is, Debian, RedHat and Slackware, have all been
updated with version 2.6 of mount.)

Using loopback means that when you completely destroy the filesystem
(as you quite possibly will), all you have to do is copy the raw
partition back and start again.

7. Preparing to write data elsewhere

You need to make sure you have a rescue partition somewhere.
Hopefully, your system has several partitions on it: perhaps a root, a
/usr, and a /home. With all these to choose from, you should have no
problem: just create a new directory on one of these.

If you have only a root partition, and store everything on that (like
me, until I can get around to repartitioning), things are slightly
more awkward. Perhaps you have an MS-DOS or Windows partition you
could use? Or you have the ramdisk driver in your kernel, maybe as a
module? To use the ramdisk (assuming a kernel more recent than
1.3.48), say the following:

# dd if=/dev/zero of=/dev/ram0 bs=1k count=2048
# mke2fs -v -m 0 /dev/ram0 2048
# mount -t ext2 /dev/ram0 /mnt

This creates a 2MB ramdisk volume, and mounts it on /mnt.

A short word of warning: if you use kerneld to automatically load and
unload kernel modules, then don't unmount the ramdisk until you've
copied any files from it onto non-volatile storage. Once you unmount
it, kerneld assumes it can unload the module (after the usual waiting
period), and once this happens, the memory gets re-used by other parts
of the kernel, losing all the painstaking hours you just spent
recovering your data.

If you have any of the new `superfloppy' removable devices, they're
probably a good choice for a rescue partition location. Otherwise,
you'll just have to stick with floppies.

The other thing you're likely to need is a program which can read the
necessary data from the middle of the partition device. At a pinch,
dd will do the job, but to read from, say, 600 MB into an 800 MB
partition, dd insists on reading but ignoring the first 600 MB. This
takes a not inconsiderable amount of time. My way round this was to
write a program which will seek to the middle of the partition. It's
called fsgrab; you can find the source package on my website
<http://pobox.com/~aaronc/tech/fsgrab-1.0.tar.gz> and it should soon
make its way onto Sunsite
<http://sunsite.unc.edu/pub/Linux/utils/disk-management/> (and
mirrors). If you want to use this method, the rest of this mini-Howto
assumes that you have fsgrab.

If none of the files you are trying to recover were more than 12
blocks long (where a block is usually one kilobyte), then you won't
need fsgrab.

If you need to use fsgrab but don't want to, it is fairly
straightforward to translate an fsgrab command-line to one for dd. If
we have

fsgrab -c count -s skip device

then the corresponding dd command is

dd bs=1k if=device count=count skip=skip

I must warn you that, although fsgrab functioned perfectly for me, I
can take no responsibility for how it performs. It was really a very
quick and dirty kludge just to get things to work. For more details
on the lack of warranty, see the `No Warranty' section in the COPYING
file included with it (the GNU General Public Licence).

8. Finding the deleted inodes

The next step is to ask the filesystem which inodes have recently been
freed. This is a task you can accomplish with debugfs. Start debugfs
with the name of the device on which the filesystem is stored:

# debugfs /dev/hda5

If you want to modify the inodes directly, add a -w option to enable
writing to the filesystem:

# debugfs -w /dev/hda5

The debugfs command to find the deleted inodes is lsdel. So, type the
command at the prompt:

debugfs: lsdel

After much wailing and grinding of disk mechanisms, a long list is
piped into your favourite pager (the value of $PAGER). Now you'll
want to save a copy of this somewhere else. If you have less, you can
type -o followed by the name of an output file. Otherwise, you'll
have to arrange to send the output elsewhere. Try this:

debugfs: quit
# echo lsdel | debugfs /dev/hda5 > lsdel.out

Now, based only on the deletion time, the size, the type, and the
numerical permissions and owner, you must work out which of these
deleted inodes are the ones you want. With luck, you'll be able to
spot them because they're the big bunch you deleted about five minutes
ago. Otherwise, trawl through that list carefully.

I suggest that if possible, you print out the list of the inodes you
want to recover. It will make life a lot easier.

9. Obtaining the details of the inodes

debugfs has a stat command which prints details about an inode. Issue
the command for each inode in your recovery list. For example, if
you're interested in inode number 148003, try this:

debugfs: stat <148003>
Inode: 148003 Type: regular Mode: 0644 Flags: 0x0 Version: 1
User: 503 Group: 100 Size: 6065
File ACL: 0 Directory ACL: 0
Links: 0 Blockcount: 12
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x31a9a574 -- Mon May 27 13:52:04 1996
atime: 0x31a21dd1 -- Tue May 21 20:47:29 1996
mtime: 0x313bf4d7 -- Tue Mar 5 08:01:27 1996
dtime: 0x31a9a574 -- Mon May 27 13:52:04 1996
BLOCKS:
594810 594811 594814 594815 594816 594817
TOTAL: 6

If you have a lot of files to recover, you'll want to automate this.
Assuming that your lsdel list of inodes to recover in is in lsdel.out,
try this:

# cut -c1-6 lsdel.out | grep "[0-9]" | tr -d " " > inodes

This new file inodes contains just the numbers of the inodes to
recover, one per line. We save it because it will very likely come in
handy later on. Then you just say:

# sed 's/^.*$/stat <\0>/' inodes | debugfs /dev/hda5 > stats

and stats contains the output of all the stat commands.

10. Recovering data blocks

This part is either very easy or distinctly less so, depending on
whether the file you are trying to recover is more than 12 blocks
long.

10.1. Short files

If the file was no more than 12 blocks long, then the block numbers of
all its data are stored in the inode: you can read them directly out
of the stat output for the inode. Moreover, debugfs has a command
which performs this task automatically. To take the example we had
before, repeated here:

debugfs: stat <148003>
Inode: 148003 Type: regular Mode: 0644 Flags: 0x0 Version: 1
User: 503 Group: 100 Size: 6065
File ACL: 0 Directory ACL: 0
Links: 0 Blockcount: 12
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x31a9a574 -- Mon May 27 13:52:04 1996
atime: 0x31a21dd1 -- Tue May 21 20:47:29 1996
mtime: 0x313bf4d7 -- Tue Mar 5 08:01:27 1996
dtime: 0x31a9a574 -- Mon May 27 13:52:04 1996
BLOCKS:
594810 594811 594814 594815 594816 594817
TOTAL: 6

This file has six blocks. Since this is less than the limit of 12, we
get debugfs to write the file into a new location, such as
/mnt/recovered.000:

debugfs: dump <148003> /mnt/recovered.000

Of course, this can also be done with fsgrab; I'll present it here as
an example of using it:

# fsgrab -c 2 -s 594810 /dev/hda5 > /mnt/recovered.000
# fsgrab -c 4 -s 594814 /dev/hda5 >> /mnt/recovered.000

With either debugfs or fsgrab, there will be some garbage at the end
of /mnt/recovered.000, but that's fairly unimportant. If you want to
get rid of it, the simplest method is to take the Size field from the
inode, and plug it into the bs option in a dd command line:

# dd count=1 if=/mnt/recovered.000 of=/mnt/resized.000 bs=6065

Of course, it is possible that one or more of the blocks that made up
your file has been overwritten. If so, then you're out of luck: that
block is gone forever. (But just imagine if you'd unmounted sooner!)

10.2. Longer files

The problems appear when the file has more than 12 data blocks. It
pays here to know a little of how UNIX filesystems are structured.
The file's data is stored in units called `blocks'. These blocks may
be numbered sequentially. A file also has an `inode', which is the
place where information such as owner, permissions, and type are kept.
Like blocks, inodes are numbered sequentially, although they have a
different sequence. A directory entry consists of the name of the
file and an inode number.

But with this state of affairs, it is still impossible for the kernel
to find the data corresponding to a directory entry. So the inode
also stores the location of the file's data blocks, as follows:

� The block numbers of the first 12 data blocks are stored directly
in the inode; these are sometimes referred to as the direct blocks.

� The inode contains the block number of an indirect block. An
indirect block contains the block numbers of 256 additional data
blocks.

� The inode contains the block number of a doubly indirect block. A
doubly indirect block contains the block numbers of 256 additional
indirect blocks.

� The inode contains the block number of a triply indirect block. A
triply indirect block contains the block numbers of 256 additional
doubly indirect blocks.

Read that again: I know it's complex, but it's also important.

Now, the current kernel implementation (certainly for all versions up
to and including 2.0.30) unfortunately zeroes all indirect blocks (and
doubly indirect blocks, and so on) when deleting a file. So if your
file was longer than 12 blocks, you have no guarantee of being able to
find even the numbers of all the blocks you need, let alone their
contents.

The only method I have been able to find thus far is to assume that
the file was not fragmented: if it was, then you're in trouble.
Assuming that the file was not fragmented, there are several layouts
of data blocks, according to how many data blocks the file used:

0 to 12
The block numbers are stored in the inode, as described above.

13 to 268
After the direct blocks, count one for the indirect block, and
then there are 256 data blocks.

269 to 65804
As before, there are 12 direct blocks, a (useless) indirect
block, and 256 blocks. These are followed by one (useless)
doubly indirect block, and 256 repetitions of one (useless)
indirect block and 256 data blocks.

65805 or more
The layout of the first 65804 blocks is as above. Then follow
one (useless) triply indirect block and 256 repetitions of a
`doubly indirect sequence'. Each doubly indirect sequence
consists of a (useless) doubly indirect block, followed by 256
repetitions of one (useless) indirect block and 256 data blocks.

Of course, even if these assumed data block numbers are correct, there
is no guarantee that the data in them is intact. In addition, the
longer the file was, the less chance there is that it was written to
the filesystem without appreciable fragmentation (except in special
circumstances).

You should note that I assume throughout that your blocksize is 1024
bytes, as this is the standard value. If your blocks are bigger, some
of the numbers above will change. Specifically: since each block
number is 4 bytes long, blocksize/4 is the number of block numbers
that can be stored in each indirect block. So every time the number
256 appears in the discussion above, replace it with blocksize/4. The
`number of blocks required' boundaries will also have to be changed.

Let's look at an example of recovering a longer file.

debugfs: stat <1387>
Inode: 148004 Type: regular Mode: 0644 Flags: 0x0 Version: 1
User: 503 Group: 100 Size: 1851347
File ACL: 0 Directory ACL: 0
Links: 0 Blockcount: 3616
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x31a9a574 -- Mon May 27 13:52:04 1996
atime: 0x31a21dd1 -- Tue May 21 20:47:29 1996
mtime: 0x313bf4d7 -- Tue Mar 5 08:01:27 1996
dtime: 0x31a9a574 -- Mon May 27 13:52:04 1996
BLOCKS:
8314 8315 8316 8317 8318 8319 8320 8321 8322 8323 8324 8325 8326 8583
TOTAL: 14

There seems to be a reasonable chance that this file is not
fragmented: certainly, the first 12 blocks listed in the inode (which
are all data blocks) are contiguous. So, we can start by retrieving
those blocks:

# fsgrab -c 12 -s 8314 /dev/hda5 > /mnt/recovered.001

Now, the next block listed in the inode, 8326, is an indirect block,
which we can ignore. But we trust that it will be followed by 256
data blocks (numbers 8327 through 8582).

# fsgrab -c 256 -s 8327 /dev/hda5 >> /mnt/recovered.001

The final block listed in the inode is 8583. Note that we're still
looking good in terms of the file being contiguous: the last data
block we wrote out was number 8582, which is 8327 + 255. This block
8583 is a doubly indirect block, which we can ignore. It is followed
by up to 256 repetitions of an indirect block (which is ignored)
followed by 256 data blocks. So doing the arithmetic quickly, we
issue the following commands. Notice that we skip the doubly indirect
block 8583, and the indirect block 8584 immediately (we hope)
following it, and start at block 8585 for data.

# fsgrab -c 256 -s 8585 /dev/hda5 >> /mnt/recovered.001
# fsgrab -c 256 -s 8842 /dev/hda5 >> /mnt/recovered.001
# fsgrab -c 256 -s 9099 /dev/hda5 >> /mnt/recovered.001
# fsgrab -c 256 -s 9356 /dev/hda5 >> /mnt/recovered.001
# fsgrab -c 256 -s 9613 /dev/hda5 >> /mnt/recovered.001
# fsgrab -c 256 -s 9870 /dev/hda5 >> /mnt/recovered.001

Adding up, we see that so far we've written 12 + (7 * 256) blocks,
which is 1804. The `stat' results for the inode gave us a
`blockcount' of 3616; unfortunately these blocks are 512 bytes long
(as a hangover from UNIX), so we really want 3616/2 = 1808 blocks of
1024 bytes. That means we need only four more blocks. The last data
block written was number 10125. As we've been doing so far, we skip
an indirect block (number 10126); we can then write those last four
blocks.

# fsgrab -c 4 -s 10127 /dev/hda5 >> /mnt/recovered.001

Now, with some luck the entire file has been recovered successfully.

11. Modifying inodes directly

This method is, on the surface, much easier. However, as mentioned
above, it cannot cope with files longer than 12 blocks.

For each inode you want to recover, you must set the usage count to
one, and set the deletion time to zero. This is done with the mi
(modify inode) command in debugfs. Some sample output, modifying
inode 148003 from above:

debugfs: mi <148003>
Mode [0100644]
User ID [503]
Group ID [100]
Size [6065]
Creation time [833201524]
Modification time [832708049]
Access time [826012887]
Deletion time [833201524] 0
Link count [0] 1
Block count [12]
File flags [0x0]
Reserved1 [0]
File acl [0]
Directory acl [0]
Fragment address [0]
Fragment number [0]
Fragment size [0]
Direct Block #0 [594810]
Direct Block #1 [594811]
Direct Block #2 [594814]
Direct Block #3 [594815]
Direct Block #4 [594816]
Direct Block #5 [594817]
Direct Block #6 [0]
Direct Block #7 [0]
Direct Block #8 [0]
Direct Block #9 [0]
Direct Block #10 [0]
Direct Block #11 [0]
Indirect Block [0]
Double Indirect Block [0]
Triple Indirect Block [0]

That is, I set the deletion time to 0 and the link count to 1 and just
pressed return for each of the other fields. Granted, this is a
little unwieldy if you have a lot of files to recover, but I think you
can cope. If you'd wanted chrome, you'd have used a graphical
`operating system' with a pretty `Recycle Bin'.

By the way: the mi output refers to a `Creation time' field in the
inode. This is a lie! (Or misleading, anyway.) The fact of the
matter is that you cannot tell on a UNIX filesystem when a file was
created. The st_ctime member of a struct stat refers to the `inode
change time', that is, the last time when any inode details were
changed. Here endeth today's lesson.

Note that more recent versions of debugfs than the one I'm using
probably do not include some of the fields in the listing above
(specifically, Reserved1 and (some of?) the fragment fields).

Once you've modified the inodes, you can quit debugfs and say:

# e2fsck -f /dev/hda5

The idea is that each of the deleted files has been literally
undeleted, but none of them appear in any directory entries. The
e2fsck program can detect this, and will add a directory entry for
each file in the /lost+found directory of the filesystem. (So if the
partition is normally mounted on /usr, the files will now appear in
/usr/lost+found.) All that still remains to be done is to work out
the name of each file from its contents, and return it to its correct
place in the filesystem tree.

When you run e2fsck, you will get some informative output, and some
questions about what damage to repair. Answer `yes' to everything
that refers to `summary information' or to the inodes you've changed.
Anything else I leave up to you, although it's usually a good idea to
say `yes' to all the questions. When e2fsck finishes, you can remount
the filesystem.

Actually, there's an alternative to having e2fsck leave the files in
/lost+found: you can use debugfs to create a link in the filesystem to
the inode. Use the link command in debugfs after you've modified the
inode:

debugfs: link <148003> foo.txt

This creates a file called foo.txt in what debugfs thinks is the
current directory; foo.txt will be your file. You'll still need to
run e2fsck to fix the summary information and block counts and so on.

12. Will this get easier in future?

Yes. In fact, I believe it already has. Kernels in the development
2.1.x series have not zeroed indirect blocks since more than six
months ago. At the beginning of December 1996, there was some talk on
the linux-kernel mailing-list of producing another 2.0.x production
kernel that also leaves indirect blocks intact on deletion. Although
as of the pre-released versions of kernel 2.0.31 this has not
happened, I suspect that it is feasible. Once Linus and the other
kernel hackers overcome this limitation in the production kernels, a
lot of my objections to the technique of modifying inodes by hand will
disappear. At the very latest, this should happen on the release of
the 2.2.x kernel series, which (according to historical kernel
development time-scales) should happen some time in the first quarter
of 1998. When this wart is corrected, it will also be possible to use
the dump command in debugfs on long files.

13. Are there any tools to automate this process?

As it happens, there are. Unfortunately, I believe that they suffer
from the same problem as the manual inode modification technique:
indirect blocks are unrecoverable. However, given the likelihood that
this will shortly no longer be a problem, it's well worth looking
these programs out now.

Someone on the net mentioned lde by Scott Heavner. To be honest, I
wouldn't recommend this as a tool for automating file recovery. It's
more like a full-screen debugfs than anything else, although it does
have some features like the ability to scan for certain types of file
or for certain file contents. It also works with the xia (does anyone
actually use this any more?) and minix filesystems, which I guess is
its major selling point these days. Version 2.3.4 is available on
Sunsite
<ftp://sunsite.unc.edu/pub/Linux/system/Filesystems/lde-2.3.4.tar.gz>
and mirrors (although it's possible there's a more recent version than
this; I found that one on an 8-month-old CD-ROM archive). lde does
have some fairly useful documentation on basic filesystem concepts, as
well as a document on how to use it for recovering deleted files.
Although I haven't used it, I suspect that my method above is better.

It sounds like the program that really works is the GNU Midnight
Commander, mc. This is a full-screen file management tool, based
AFAIK on a certain MS-DOS program commonly known as `NC'. mc supports
the mouse on the Linux console and in an xterm, and provides virtual
filesystems which allow tricks like cd-ing to a tarfile. Among its
virtual filesystems is one for ext2 undeletion. It all sounds very
handy, although I must admit I've never used the program myself -- I
prefer good old-fashioned shell commands. Apparently one must
configure the program with the --with-ext2undel option; you'll also
need the development libraries and include files that come with the
e2fsprogs package. I gather that once the program is built, you can
tell it to cd undel:dev/hda5/, and get a `directory listing' of
deleted files.

The latest non-development version is probably 4.0; as with the kernel
itself, development versions are not recommended to non-hackers. The
list of (over 70) download sites is available on the Midnight
Commander 4 website <http://mc.blackdown.org/mc4/>, or try the
official ftp site
<ftp://ftp.nuclecu.unam.mx/linux/local/mc-4.0.tar.gz> (which if memory
serves is rather slow).

14. Colophon

I intend to produce regular updates to this document as long as I have
both enough time to do it, and something interesting to say. This
means that I am eager to hear comments from readers. Could my writing
be clearer?. Can you think of something that would make matters
easier? Is there some new tool that does it all automatically? Who
did kill JFK?

Whatever. If you have something to say, about this document or any
other subject, drop me a line on <[email protected]>.

15. Credits and Bibliography

`If I have seen farther than others, it is because I was
standing on the shoulders of giants.' (Isaac Newton)

Much of this mini-Howto was derived from a posting in the
comp.os.linux.misc newsgroup by Robin Glover <[email protected]>.
I would like to thank Robin for graciously allowing me to rework his
ideas into this mini-Howto.

Some bibliographic references:

� Frisch, �leen (1995), Essential System Administration, second
edition, O'Reilly and Associates, Inc., ISBN: 1-56592-127-5.

� Glover, Robin (31 Jan 1996), HOW-TO : undelete linux files
(ext2fs/debugfs), comp.os.linux.misc Usenet posting.

� Peek, Jerry, Tim O'Reilly, Mike Loukides et al (1993), UNIX Power
Tools, O'Reilly and Associates, Inc./Random House, Inc., ISBN:
0-679-79073-X.

16. Legalities

All trademarks are the property of their respective owners.
Specifically:

� MS-DOS and Windows are trademarks of Microsoft
<http://www.microsoft.com/>.

� UNIX is a trademark of the Open Group <http://www.open.org/>.

� The trademark status of the name Linux is currently being contested
by lawyers. A certain Walter R. Della Croce has made an allegedly
false trademark registration for the term. Further information on
the Linux trademark issue is available from the Linux Mall
<http://www.linuxmall.com/announce/>.

This document is Copyright � 1997 Aaron Crane <[email protected]>. It
may be freely redistributed in its entirety, including the whole of
this copyright notice, but may not be changed without permission from
either the author or the Linux Documentation Project Coordinator.
Dispensation is granted for copying small verbatim portions for the
purposes of reviews or for quoting; in these circumstances, sections
may be reproduced in the presence of an appropriate citation but
without this copyright notice.

The author requests but does not require that parties intending to
sell copies of this document, whether on computer-readable or human-
readable media, inform either him or the Linux HOWTO Coordinator of
their intentions.

The Linux HOWTO Coordinator is Greg Hankins <[email protected]>.