Mail2News Mini-Howto

Mail2News Mini-Howto
Rick Dean <[email protected]>
v2.0, 1 July 1999

This document describes how to feed a mailing list to a news server.

1. Copyright, Distribution, etc.

1.1. What is this?

This document describes how to feed a mailing list to a news server
using a Linux box. It is called a mini-HOWTO, specifically the
Mail2news mini-HOWTO.

1.2. Copyright and such

Copyright (c) 1999, Rick Dean.
Copyright (c) 1996, Robert Hart.

The authors retain their copyright of this document. You are hereby
granted permission to redistribute this document in whole or in part
as long as it includes this copyright notice. Commercial
redistribution is allowed and encouraged. All translations or
derivative works of this document must be covered under this copyright
notice, and without additional restrictions on distribution. This
arrangement is also known as a copyleft.

This copyright notice, itself, is hereby placed in the public domain.
You may copy it without atribution.

1.3. Where can I get this HOW-TO?

The latest version of this document is available at
http://fdd.com/howto/ <http://fdd.com/howto/>.

Many Linux distributions include mini-HOWTOs in the /usr/doc/HOWTO
directory.

This mini-HOWTO like most Linux mini-HOWTO's was written initially is
SGML. This allows the text to be automatically translated to many
formats including text, HTML, PostScript, etc. Those other formats
are available somewhere.

More information about Linux documentation can be found at the Linux
Documentation Project <http://www.redhat.com/mirrors/LDP/>, and so
many other places.

1.4. Document history

� Version 1.0 was written by Robert Hart <[email protected]> in 1996

� Version 2.0 was written by Rick Dean <[email protected]> in 1999

2. Overview

2.1. What? Why? and some definitions

This document describes how to feed a mailing list to a news server
using a Linux box. It is called a mini-HOWTO, specifically the
Mail2news mini-HOWTO.

A mailing list (also known as a remailer), is an address where e-mail
will be resent to a list of other addresses. This is useful for
colaboration of geographically disperse groups. Many standards bodies
like the working groups of the IETF use mailing lists.

Unfortunately, if one is subscribed to several mailing lists, one's
inbox may be routinely flooded. Furthermore, some companies (such as
3Com) specify which e-mail client (such as Lotus Notes) their
employees must suffer with. Redirecting these e-mails to a news
server frees people to choose a news reader and utilize refined
features specifically designed for the task (of deriving signal from
noise).

News servers started on the Internet long ago, many years before the
WWW. They (and the news reader clients) have features such as...

� threading - responses are grouped with the e-mails they refer to

� read indication - the computer keeps track of what you have seen
(even across newsgroups!) so you don't have to.

� subject kill - mark threads as read by thread.

� archiving

� segregation by newsgroup (i.e. mailing list)

� a quick standardized method subscribing/unsubscribing

� and more..

A big focus of news servers is sharing news between servers. The
largest of these groups became known as USENET. This mini-HOWTO does
not address that. You could share the newsgroups created with this
mini-HOWTO on your own, but you will live just fine without it. Like
a web server, ubiquitous Internet connectivity has made centralized
news servers acceptable. Furthermore, recent benchmarks have shown
single-processor Linux boxes can handle 1300+ HTTP hits per second, so
scalability is a minor issue.

Athough you do not need to own the mailing list to use mail2news, it
is a good idea to own the news server.

2.2. Assumed environment

This document assumes you are using Linux, but other Unices are nice.
Currently only sendmail (for a mail delivery agent) is described, but
as qmail grows in popularity (in part because it is easier to
configure). Hopefully someone with submit configuration notes for it
too. For a news server, this document describes innd. It is pretty
dominant as news servers go, but any NNTP compliant one should work.
A bit of glue called mail2news.pl is a perl script, thus you need the
Perl interpreter, but it is very common and probably already
installed. Finally, I assume you are running all this (except the
mailing list remailer) on one machine. Dividing it up is left as an
exercise for the reader. :-)

At the time of this writing this mini-HOWTO was only tested against a
RedHat-6.0 distribution. As a good computer scientist, you should not
believe anything works until it has been specifically tested. Any
feedback or notes relating to other distributions would be welcomed by
the author.

For most of this mini-HOWTO you will need root access unless otherwise
specified.

2.3. Methodology

This mini-HOWTO is presented backwards, as this is the easiest way to
build and debug it. Backwards means we start with the newsreader and
work upstream to the mailing list remailer, the opposite direction of
normal data flow. This systems uses several hairy pieces (like
sendmail and innd) which are sizeable mini-HOWTO's in thier own right.

2.4. Not covered

This mini-HOWTO does not cover...

� setting up a mailing list remailer.

� exchanging news between news servers.

� a complete list of things not covered.

Please do not e-mail me about these subjects (or SPAM).

3. The news reader

3.1. Netscape

Netscape comes with an integrated news reader. The easy way to
subscribe to a group is type in (or click on) a link like

news://fuji.sfour.com/ietf.confctrl

Once you have subscribed, you only need to go to the message center.
This can be done by clicking on the small talk balloons icon in the
bottom right of a browser window.

3.2. Free Agent

Free Agent is a wonderful news reader by Forte for Windoze. See
http://www.forteinc.com/agent/freagent.htm

3.3. trn and friends

The old command line newsgreaders like trn and tin are a good standby.
Be sure to set the environment variable NNTPSERVER first. For
example...

export NNTPSERVER=fuji.sfour.com
trn

4. The news server

4.1. Installing the news server

You need to install a news server. I used inn but others are
available (somewhere).

Preferably just check the "News Server" box during your initial
install, but alternatively if your distribution uses RPM (RedHat
package manager), then use something like...

rpm -i inn-2.2.9.i386.rpm

4.2. Running the newsserver

To manually start or stop the news server, use a command like

/etc/rc.d/init.d/innd start

or

/etc/rc.d/init.d/innd stop

or

/etc/rc.d/init.d/innd restart

To have the news server start at boot you could add a command like
this to end of /etc/rc.d/rc.local, but that is not the best. Many
distributions have a graphical tool for choosing which daemons run.
You can also try the command line program

setup

or

chkconfig --add innd

4.3. Creating the newsgroup

innd is pretty picky about permissions and ownership. For much of the
news config you will need to be the user news. To become this user
from root...

su - news

Using ctlinnd, create the newsgroup on your news server. Remember,
the newsgroup will be local, so start it with a distinctive name so
you can filter it out from your news distributions if you do that
stuff. I shamelessly named my newsgroup ietf.confctrl. The words
from left to right go from less to more specific.

You also need to tell innd that the group is moderated (by using
ctlinnd). Indicating a moderated group is done by specifying m to the
newgroup command. For example...

ctlinnd newgroup ietf.confctrl m [email protected]

The newsgroup is set up as a moderated group, as this allows us to
take advantage of the email capabilities of innd. Any messages posted
to a moderated group are not immediately submitted to the group.
Instead, messages are emailed to the moderator of the group. In our
example [email protected] is the address which is resent by the
remailer.

If you are sharing news with other servers, remember to edit your
newsfeeds so that this group is not not distributed (unless you
specificaly wish this to occur).

4.4. Unrestricting access

By default the news server, doesn't let any clients read news, so I
needed to disable the user authentication of innd. This was done in
the /etc/news/nnrp.access. Check out the nnrp.access man page to
learn the syntax of this file. I changed the first non-comment line
to ...

*:Read Post:::*

If you want a username/password, fill in the 3rd and 4th (colin
separated) fields. For more information on the syntax, check the man
page...

man nnrp.access

4.5. Changing permissions of /usr/bin/rnews

I had to change the permissions of /usr/bin/rnews. It was not world
read/executable, but sendmail runs scripts as nobody.

chmod a+rx /usr/bin/rnews

4.6. Testing article posting

If you copy the following article to a file named rick.article...

______________________________________________________________________
Path: rick
From: [email protected]
Message-ID: <[email protected]>
Subject: test
Date: Mon, 12 Jul 1999 00:48:49 -0500 (CDT)
Newsgroups: ietf.confctrl
Approved: [email protected]
NNTP-Posting-Host: localhost
Organisation: (mail2news gateway)

test
______________________________________________________________________

Then you should be able to post a file with...

/usr/bin/rnews -r localhost <rick.article

4.7. Did it work?

You don't need to wait for the article to show up as unread, just look
at the /var/spool/news/articles/ subdirectories for files being
created.
4.8. Increasing expiration times

You may want to increase the expiration time for articles of your new
newsgroup. In my case I wanted them never to expire, so I added the
following line....

ietf*:A:never:never:never

....to the /etc/news/expire.ctl file.

To learn more about the syntax of this file type...

man expire.ctl

5. The posting script & perl

5.1. What is Perl?

Perl stands for Practical Extension and Report Language. It is very
popular for small scripts which manipulate text which is exactly what
we need.

Perl is installed by default on almost every Unix system.

5.2. Location of perl

If you perl intepreter is in an unusual place (not /usr/bin/) then you
will have to modify the first line of the script. If this line is
wrong, on my 2.2 kernel system I get "bash:
/usr/local/bin/mail2news.pl: No such file or directory" Can we please
change this to "bash: /usr/local/bin/mail2news.pl: Interpreter not
found. Check first line of script." ?

5.3. The mail2news.pl script

______________________________________________________________________
#!/usr/bin/perl

($program = $0) =~ s%.*/%%;

#( $version ) = $] =~ /(\d+\.\d+).*\nPatch level/;
#die "$program: requires at least version 3 of perl\n"
# if $version < 3;

# $news_poster_program = "/usr/bin/inews";
# $news_poster_options = "-h -o \"mail2news gateway\"";
$news_poster_program = "/usr/bin/rnews";
$news_poster_options = "-r localhost";
$postinghost = "localhost";

if ($#ARGV < 0) {
# $newsgroup = "test";
# we'll expect the newsgroup line in the body
} elsif ($#ARGV == 0) {
$newsgroup = $ARGV[0];
} else {
die "usage: $program [newsgroup]\n";
}

# in case inews dumps core or something crazy
$SIG{'PIPE'} = "plumber";
sub plumber { die "$program: \"$news_poster_program\" died prematurely!\n"; }

open (INEWS, "| $news_poster_program $news_poster_options") ||
die "$program: can't run $news_poster_program\n";

# header munging loop
while (<STDIN>) {
last if /^$/;

# transform real from: line back to icky style
s/^From:\s+(.*) <(.*)>/From: $2 ($1)/;

s/Message-Id/Message-ID/;

# transform from_ line to path header; also works locally
s/^From\s+(\S+)@(\S+).*/Path: $2!$1/
|| s/^From\s+(\S+)[^@]*$/Path: $1\n/;

print INEWS
# if /^(Date|From|Subject|Path|Newsgroups|Organization|Message-ID):/i;
if /^(Date|From|Subject|Path|Newsgroups|Message-ID):/i;
$saw_subject |= ( $+ eq 'Subject' );

$saw_msgid |= ( $+ eq 'Message-ID' );

# $saw_newsgroup |= ( $+ eq 'Newsgroups' );
}

warn "$program: didn't expect newsgroup in both headers and ARGV\n"
if $newsgroup && $saw_newsgroup;

die "$program: didn't get newsgroup from either headers or ARGV\n"
unless $newsgroup || $saw_newsgroup;

$approved = $newsgroup;
$approved =~ s/\./'-'/eg;

($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
$madeupid = "\<$year$mon$mday.$hour$min$sec.$$\@kepler.hedland.edu.au\>";

printf INEWS "Newsgroups: %s\n", $newsgroup if $newsgroup;
printf INEWS "Approved: %s\@kepler.hedland.edu.au\n", $approved;
print INEWS "Subject: Untitled\n" unless $saw_subject;
printf INEWS "Message-ID: %s\n", $madeupid unless $saw_msgid;
printf INEWS "NNTP-Posting-Host: %s\n", $postinghost;
print INEWS "Organisation: (mail2news gateway)\n";
print INEWS "\n";

print INEWS while <STDIN>; # gobble rest of message

close INEWS;
exit $?;
______________________________________________________________________

I saved the script in /usr/local/bin (and will use this path
throughout the HOWTO).

Be sure to make the script executable by all, but not writable by
group or other. Sendmail is picky.

chmod a+x /usr/local/bin/mail2news.pl
chmod go-w /usr/local/bin/mail2news.pl

or

chmod 555 /usr/local/bin/mail2news.pl

for short.

5.4. How do I know if the script is running?

I tested this script by changed my news poster from /usr/bin/rnews to
/bin/cat. I then saved an e-mail send to myself in a file. Finally I
ran the mail2news.pl on the saved mail and captured the output to a
file.

/usr/local/bin/mail2news.pl ietf.confctrl </tmp/savedMailFile >/tmp/article

5.5. What is with the Aussie?

Yes, the output of your posting script should contain the e-mail
address of an austrailian. My guess is that the Austrailian's address
is trusted address in your news configuration (althogh I could not
find it in mine) for approving moderated postings.

5.6. What do you mean $PATH?

If the mail2news.pl script is not in my path, I get the error bash:
mail2news.pl: command not found. You will need to either add this
directory to your path

PATH=$PATH:/usr/local/bin

(which only works for the current login) or give an absolute path when
you run the script

/usr/local/bin/mail2news.pl

To view your current path type

echo $PATH

6. The local mailer daemon

6.1. Enabling alias files

This document only describes modifying sendmail. Many people prefer
qmail.

Out of the box, sendmail does not come with the "pipe to program"
feature enabled. Rightly so, this feature is a security risk. It
enambles users to have their mail fed to a program instead of appended
to a file. (Users can configure this in thier .forward file.)

Please don't believe me. Test your own system first. Much of this
chapter might not be needed by you.

6.2. Documentation

IMHO, sendmail installs with woefully insufficient documentation.

http://www.sendmail.org/ ca/email/doc/op-sh-5.html describes the
syntax of sendmail.cf. I hope you don't have to use it.

This is a description of the features you are trashing in sendmail.mc
http://www.sendmail.org/m4/features.html

Sendmail has a man page...

man sendmail

6.3. Recompiling your sendmail.cf file

To get the "pipe to program" stuff in the aliases file to work you
need to modify the default sendmail.cf (by modifying the sendmail.mc)
so that the restricted shell is not used. I suppose the proper
solution would be to add the one program to the restricted shell list,
but their was no man page on smrsh. Strangely, uncommenting the smrsh
feature didn't work, I needed to change the shell from /usr/sbin/smrsh
to /bin/bash. Yeah, this is slightly risky, but it was not an issue
on my machine. Without this change I kept getting a "Service
unavailable" error message in the /var/log/maillog file.

The header of /etc/senmail.mc of RedHat-6 has a bug. The proper
command line is...

m4 /etc/sendmail.mc >/etc/sendmail.cf

You need to do this when you change sendmail.mc. Hopefully, RedHat
will extend the super cool Makefile idea in /etc/mail You will need to
install sendmail-cf.something.rpm first. e.g. ...

rpm -i sendmail-cf-8.9.3-10.i386.rpm

Whenever you modify the sendmail.cf file, you should restart
sendmail...

/etc/rc.d/init.d/sendmail restart

6.4. Creating a mail alias

Instead of creating a new user account, we will only create an alias.

When modifying the /etc/aliases file, the double quotes are required.
There cannot be a space between the first double quotes and the |
(pipe) character, or sendmail will complain "User unknown"

Add a line like ...

confctrl: "| /usr/local/bin/mail2news.pl ietf.confctrl "

Whenever you modify the /etc/aliases file you need to notify sendmail.

sendmail -bi

6.5. Did it work?

Check the /var/log/maillog to see if it worked, or for error messages.
I found it useful to open up another terminal (ssh) window to watch
the log with

tail -f /var/log/maillog

6.6. Unprivledged

If you are having trouble, and create an e-mail alias with a different
(simpler) target program to test it, remember that sendmail runs the
program as an unprivledged user, who probably doesn't have privledges
to write anywhere except globally writable directories such as /tmp.

7. The mailing list remailer

7.1. Subscribing

The method of subscribing to each e-mail list is different, although
most involve sending some kind of e-mail message. Please be very
careful not to send to the replicating address of the mailgroup. This
is a sure way to annoy the members of the list. Most remailers (but
not all) have a separate address for subscribing. In confctrl, the
administration address is [email protected]

Some mailing lists do not let you subscribe an address you are not
mailing from. In this case you will need to forge the return address
of an e-mail.

When you successfully subscribe you should (hopefully) see a welcome
message in the newsgroup.

7.2. Forging mail

Forgeing mail is easiest to do with an old Netscape client. It would
accept anything for a return address. Fortunately, modern browsers
require you to retreive mail successfully for an address before they
will let you send. Thus you are required to fall back the tried and
true method....telnet.

7.2.1. Looking up MX hosts

When you speficy a web page, your browser does a DNS lookup to convert
the domain name into an IP address. This is a lookup of an "A"
record. (Also "CNAME" records are considered.) When sending mail a
"MX" record is looked up. If this is missing a "CNAME" or "A" one is
used. Thus, to forge mail you need to do an MX lookup.

dig mx isi.edu

will give...

; <<>> DiG 8.2 <<>> mx isi.edu
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 3, ADDITIONAL: 4
;; QUERY SECTION:
;; isi.edu, type = MX, class = IN

;; ANSWER SECTION:
isi.edu. 1D IN MX 0 tnt.isi.edu.
isi.edu. 1D IN MX 10 venera.isi.edu.

;; AUTHORITY SECTION:
isi.edu. 1D IN NS venera.isi.edu.
isi.edu. 1D IN NS ns.isi.edu.
isi.edu. 1D IN NS east.isi.edu.

;; ADDITIONAL SECTION:
tnt.isi.edu. 1D IN A 128.9.128.128
venera.isi.edu. 1D IN A 128.9.176.32
ns.isi.edu. 1D IN A 128.9.128.127
east.isi.edu. 1D IN A 38.245.76.2

;; Total query time: 448 msec
;; FROM: fdd.com to SERVER: default -- 127.0.0.1
;; WHEN: Sun Jul 25 15:49:32 1999
;; MSG SIZE sent: 25 rcvd: 182

thus you would use tnt.isi.edu.

7.2.2. SMTP

Mail is delivered using the Simple Mail Transport Protocol (SMTP).
Like most good Internet protocols, it is ASCII based to make
troubleshooting and development easier. I will not explain
everything, but simply give an example. Hopefully, this is enough.

The protocol is line oriented. Each email as specified in RFC822, is
composed of headers and body which are separated by the first blank
line (no characters, not even spaces). SMTP specifies the end of an
e-mail with a line containing only a period.

halyard$ telnet tnt.isi.edu 25
Trying 128.9.128.128...
Connected to tnt.isi.edu.
Escape character is '^]'.
220 tnt.isi.edu ESMTP Sendmail 8.8.7/8.8.6; Sun, 25 Jul 1999 14:01:25 -0700 (PDT)
helo isi.edu
250 tnt.isi.edu Hello rick@node-d8e9822 [216.233.8.34] (may be forged), pleased to meet you
mail from:<[email protected]>
250 <[email protected]>... Sender ok
rcpt to:<[email protected]>
250 <[email protected]>... Recipient ok
data
354 Enter mail, end with "." on a line by itself
From:<[email protected]>
Subject: help

help