Re: Changes to the mailinglist setup

Hi

as the following affects all people subscribed to a list on
lists.spi-inc.org I crosspost a bit. If anyone really replies please
select one list, either spi-private or spi-general, depending on how you
want to flame me, to reply to.
Thanks.

Now, after we changed the MTAs installed on SPIs machines from Exim4 to
Postfix, together with some filters, which already seems to have dropped
the amount of spam received[1], we just did the final step to close our
lists for any spam.

[1] According to listmaster it went down to one every few days.

Now all SPI lists are closed for postings from unsubscribed members.
Every mail from a non-member now gets into the moderation queue. This
queue is processed by a listmod bot, which will ask spamassassin and
bogofilter. Based on their judgement it will either accept or delete the
mail. In case both are unsure about its ham/spam status the bot will ask
humans (our listmoderators) for help. Whatever the human decided (ham/spam),
the bot will learn from it, to get better.

This means that non-member posts get a delay of only ~5 minutes[2] added
before it reaches the list. It also means that it greatly reduces the
load of the humans doing listadmin stuff, as they only need to look at
mails where the bot is unsure.

[2] if the bot can decide himself. If a human is needed then it is of
course a bit longer but shouldnt ever be longer than 24hours.

There is a second effect with this setup - if people are subscribed with
one address but post with another. Of course they get into the
moderation queue and need to wait for the bot to look at their mail.
If that happens and one wants to avoid the 5 minute delay there are two
options:
- post from the subscribed address only, or
- subscribe the different addresses you want to post from, but set all
except one as "nomail" to not get duplicates.

This way of running lists is already used by at least the DebConf lists,
Backports.org lists and the lists run by OFTC and works damn well.

--
bye Joerg
<Fubak> /msg NickServ IDENTIFY arschloch
<codebreaker> /msg nickserv ghost Fubak arschloch
-!- Fubak has quit [Nick collision from services.]

On Sat, Oct 21, 2006 at 07:09:09PM +0200, Joerg Jaspert wrote:
>
> Now, after we changed the MTAs installed on SPIs machines from Exim4 to
> Postfix, together with some filters, which already seems to have dropped
> the amount of spam received[1], we just did the final step to close our
> lists for any spam.

Hi Joerg,

If you have time, I'd be very interested in the specific filter setup
you're using, and also how postfix has helped out. I switched from
Postfix to Exim4 awhile back to help with the spam situation, so maybe
there's something I'm not aware of.

If you'd rather not, that's fine too.

-- John

On 10814 March 1977, John Goerzen wrote:

> If you have time, I'd be very interested in the specific filter setup
> you're using, and also how postfix has helped out. I switched from
> Postfix to Exim4 awhile back to help with the spam situation, so maybe
> there's something I'm not aware of.

Well, we changed to postfix because 3/4 of the SPI admins can work with
postfix, while only 2/4 can work with exim4, so thats the main reason
for the switch.

That this new setup now catches more spam seems to be a side effect
reported by a listmaster who needed to do way less moderation after the
change than he needed to do before, with the few closed lists we had
then. I personally don't understand much of exim, so cant really judge
the setup we had, I only know we had a shitload of spam on the lists
(and complains from board members about it), and if one looks in the
archives for lists that have been open until today, like board, you see
that at the date the MTAs switched the spam goes back to nearly 0.....

Now, my setup - well. It has sender verify, greylisting[1], virus and
spamscans[2], reject with some rbl lists[3] and then have different
accept/reject lists at different stages of the smtp protocol where we
can also intercept. There is also a regexp list for body and for header
checks. And then the usual "non-fqdn recipient/sender/hostname, etc"
stuff.

To not have the backup MX be an open hole for our mailsetup that one
knows about all addresses that are valid for the domains it "backs
up". Thats done by a simple perl script that updates its address list
every 15 minutes. Based on that the backup MX has the same checks, ie
also spam, virus, etc. To avoid double filtering if a mail gets in via
the backup MX it uses a tls connection to the primary MX, with a known
certificate which gets it to bypass all other checks on the primary.

It may not be the most perfect setup of the world but its working very
well.

[1] with automated whitelisting after you got 5 mails sent in
[2] not done during smtp time. We accept mail but kill silently if it
has a spamassassin score above 10 or is a virus
[3] [bogusmx|dsn].rfc-ignorant.org, blackhole.securityusage.com,
sbl-xbl.spamhaus.org, relays.ordb.org, opm.blitzed.org,
list.dsbl.org, ie. the more sane ones

--
bye Joerg
dvdbackup (0.1.1-7) unstable; urgency=medium
.
* The wiki-wacky-oaxtepec release

On Sun, Oct 22, 2006 at 01:36:06AM +0200, Joerg Jaspert wrote:
> [1] with automated whitelisting after you got 5 mails sent in
> [2] not done during smtp time. We accept mail but kill silently if it
> has a spamassassin score above 10 or is a virus
> [3] [bogusmx|dsn].rfc-ignorant.org, blackhole.securityusage.com,
> sbl-xbl.spamhaus.org, relays.ordb.org, opm.blitzed.org,
> list.dsbl.org, ie. the more sane ones

Thanks for the helpful information. I'll be testing this on my own
server.

Quick question: opm.blitzed.org says that it was discontinued in May,
and I couldn't find securityusage.com. Do you have any further info on
those?

Thanks,

-- John

On 10815 March 1977, John Goerzen wrote:

>> [1] with automated whitelisting after you got 5 mails sent in
>> [2] not done during smtp time. We accept mail but kill silently if it
>> has a spamassassin score above 10 or is a virus
>> [3] [bogusmx|dsn].rfc-ignorant.org, blackhole.securityusage.com,
>> sbl-xbl.spamhaus.org, relays.ordb.org, opm.blitzed.org,
>> list.dsbl.org, ie. the more sane ones
> Thanks for the helpful information. I'll be testing this on my own
> server.

> Quick question: opm.blitzed.org says that it was discontinued in May,
> and I couldn't find securityusage.com. Do you have any further info on
> those?

Both dead, yeah. I could go and check what I use, again. Luckily it
doesnt hurt[1] (and that makes one a bit lazy). :)

[1] except you are dumb and use spamcop, that always hurts. :)

--
bye Joerg
> But i don't think that we talk a lot, as far as i can see, you live in
> the USA.
Australia. Only minor details like timezone and hemisphere but pretty
much the same. TZ is UTC+10

On Sun, Oct 22, 2006 at 01:36:06AM +0200, Joerg Jaspert wrote:
> Well, we changed to postfix because 3/4 of the SPI admins can work with
> postfix, while only 2/4 can work with exim4, so thats the main reason
> for the switch.
>
> That this new setup now catches more spam seems to be a side effect
> reported by a listmaster who needed to do way less moderation after the
> change than he needed to do before, with the few closed lists we had
> then. I personally don't understand much of exim, so cant really judge
> the setup we had, I only know we had a shitload of spam on the lists
> (and complains from board members about it), and if one looks in the
> archives for lists that have been open until today, like board, you see
> that at the date the MTAs switched the spam goes back to nearly 0.....

Since you admit you don't understand much of Exim, please don't conflate the
issue of letting spam through filters with the choice of MTA :P

The previous setup actually did have most of the things that you mention
below, and in-session scanning at that, but it clearly didn't have the same
level of filtering ability - AFAIK, neither me nor Wichert spent any
particular time training the Bayesian classifier in the recent months, we
didn't do any additions to the list of RBLs either, our SA was 3.0 still I
think so we didn't have sa-update (which updates the SA custom rules from
upstream), and we didn't have any special Mailman anti-spam provisions like
the ones you added.

Postfix is now used because Joerg knows Postfix, and Joerg is doing the work.
Let's acknowledge that, and drop the unqualified Exim bashing, mmmkay?

--
2. That which causes joy or happiness.

> Postfix is now used because Joerg knows Postfix, and Joerg is doing the work.
> Let's acknowledge that, and drop the unqualified Exim bashing, mmmkay?

Oh good lord. There wasn't any Exim bashing going on. They guy readily
admitted that he didn't know Exim and that it seemed to make a
difference when he changed setups. Likely Exim wasn't configured as well
or thoroughly as the new Postfix installation.

However I am glad to see us move to a MTA that more than just Debian
people will know ;)

Joshua D. Drake

On Sun, Oct 22, 2006 at 07:32:40AM -0700, Joshua D. Drake wrote:
> >Postfix is now used because Joerg knows Postfix, and Joerg is doing the
> >work.
> >Let's acknowledge that, and drop the unqualified Exim bashing, mmmkay?
>
> Oh good lord. There wasn't any Exim bashing going on. They guy readily
> admitted that he didn't know Exim and that it seemed to make a
> difference when he changed setups. Likely Exim wasn't configured as well
> or thoroughly as the new Postfix installation.

Sorry, I read more of that kind of talk on our alias and I'm a bit tired of
it, it's pointless. I simply don't believe in throwing around any sort of
assessment in relation to something I don't know about. For example, you
won't ever hear me say anything much about Oracle, because I have but a
cursory knowledge of it, even if I do seem to choose other databases over it.

> However I am glad to see us move to a MTA that more than just Debian
> people will know ;)

All four of us at admin@ are Debian people, so that didn't really change :)

--
2. That which causes joy or happiness.

Joerg Jaspert <joerg(at)debian(dot)org> wrote:
> [...], reject with some rbl lists[3] [...]
> [3] [bogusmx|dsn].rfc-ignorant.org, blackhole.securityusage.com,
> sbl-xbl.spamhaus.org, relays.ordb.org, opm.blitzed.org,
> list.dsbl.org, ie. the more sane ones

More sane perhaps, but still insane. Listing major European ISP
outgoing mail relays still seems a frequent event on at least one of the
"Real-time" Blackhole Lists. At least spamcop isn't being used. :-/

RBLs should never trigger a reject, but can be good input to a scoring
system. If one rejects on a RBL, you almost may as well reject on a
RNG. Please can SPI's RBL use be demoted to a score, not a reject?

Regards,
--
MJ Ray - see/vidu http://mjr.towers.org.uk/email.html
Somerset, England. Work/Laborejo: http://www.ttllp.co.uk/
IRC/Jabber/SIP: on request/peteble

On 10816 March 1977, MJ Ray wrote:

>> [...], reject with some rbl lists[3] [...]
>> [3] [bogusmx|dsn].rfc-ignorant.org, blackhole.securityusage.com,
>> sbl-xbl.spamhaus.org, relays.ordb.org, opm.blitzed.org,
>> list.dsbl.org, ie. the more sane ones
> More sane perhaps, but still insane. Listing major European ISP
> outgoing mail relays still seems a frequent event on at least one of the
> "Real-time" Blackhole Lists. At least spamcop isn't being used. :-/

Im currently using (and this time using copy&paste from config, not
typing from memory)
reject_rhsbl_sender bogusmx.rfc-ignorant.org
reject_rhsbl_sender dsn.rfc-ignorant.org
reject_rhsbl_sender blackhole.securitysage.com
reject_rhsbl_client blackhole.securitysage.com
reject_rbl_client sbl-xbl.spamhaus.org
reject_rbl_client relays.ordb.org
reject_rbl_client list.dsbl.org

and cant remember them having added whole major ISPs. Spamcop yes, thats
bullshit, but noone sane uses that.

> RBLs should never trigger a reject, but can be good input to a scoring
> system. If one rejects on a RBL, you almost may as well reject on a
> RNG. Please can SPI's RBL use be demoted to a score, not a reject?

Well, lets look at the mail statistic for yesterday. (Keep in mind that
we are MX for spi-inc.org, lists.spi, members.spi, fresco.org,
lists.fresco and also emdebian.org, so its not all SPI only)

We have 483 valid mails received.
In the same timeframe we rejected 3892 mails at smtp stage.
That makes 2598k bytes from 119 senders. Our smtpd had 3414 connections
from 811 hosts. The reject summary is (some are 4XX, some are 5XX):

Helo command rejected: Invalid name (total: 31)
Helo command rejected: You are obviously using a bogus helo/ehlo (total: 5)
(Where bogus ehlo means they pretend to be chic.spi-inc.org...)
Helo command rejected: need fully-qualified hostname (total: 498)
Recipient address rejected: Greylisted for [...] seconds (total: 333)
Recipient address rejected: Multi-recipient bounce (total: 2)
Recipient address rejected: User unknown in virtual alias table (total: 44)
Relay access denied (total: 22)
Sender address rejected: Domain not found (total: 99)
Sender address rejected: need fully-qualified address (total: 1)
Sender address rejected: undeliverable address (total: 280)
Sender address rejected: unverified address (total: 1935)
blocked using bogusmx.rfc-ignorant.org (total: 21)
blocked using dsn.rfc-ignorant.org (total: 37)
blocked using list.dsbl.org (total: 14)
blocked using relays.ordb.org (total: 1)
blocked using sbl-xbl.spamhaus.org (total: 565)
body regexp checks blocked 2
header regexp checks blocked 2

So the rbl lists got us rid of 638 mails. Now, lets look at who got
rejected. If one looks at the from/to/helo triple in the log I cant see
anything valid blocked currently. There are some froms where you could
think its ok, but then either the to or helo contains such random
strings that it makes itself invalid.

So no, right now Im not changing it. Im only using a limited set of RBLs
anyway that seem to be sane enough, not bullshit like spamcop. If you
can show evidence that one of those is broken then I will drop it. (Or
move it to a spamassassin check.)

--
bye Joerg
[2.6.15.4 direkt nach 2.6.15.3]
<HE> Linus muss Gentooler hassen.
<formorer> wieso?
<HE> Naja, die dürften ihre optimierten Kernel gerade fertig gebaut
haben und müssen jetzt aus prompter Versionitis auf das
Ausprobieren verzichten und den neuen kompilieren...

On Mon, Oct 23, 2006 at 09:36:05AM +0100, MJ Ray wrote:
> Joerg Jaspert <joerg(at)debian(dot)org> wrote:
> > [...], reject with some rbl lists[3] [...]
> > [3] [bogusmx|dsn].rfc-ignorant.org, blackhole.securityusage.com,
> > sbl-xbl.spamhaus.org, relays.ordb.org, opm.blitzed.org,
> > list.dsbl.org, ie. the more sane ones
>
> More sane perhaps, but still insane. Listing major European ISP
> outgoing mail relays still seems a frequent event on at least one of the
> "Real-time" Blackhole Lists. At least spamcop isn't being used. :-/

That doesn't mean they all are that way.

This looks like a very sane setup. Don't forget that it is far cheaper
resource-wise to drop them during the SMPT conversation rather than with
spamassassin later.

> RBLs should never trigger a reject, but can be good input to a scoring

I have been using the Spamhaus RBL for *years* and have yet to have had
a false positive with it.

And this on systems that process far more mail than SPI.

I think Joerg has built a very solid setup, and commend him on it.

-- John

[John Goerzen]
> I have been using the Spamhaus RBL for *years* and have yet to have
> had a false positive with it.
>
> And this on systems that process far more mail than SPI.

What method do you use to detect false positives with such large
amount of email?

Friendly,
--
Petter Reinholdtsen

On Tue, Oct 24, 2006 at 11:01:19AM +0200, Petter Reinholdtsen wrote:
>
> [John Goerzen]
> > I have been using the Spamhaus RBL for *years* and have yet to have
> > had a false positive with it.
> >
> > And this on systems that process far more mail than SPI.
>
> What method do you use to detect false positives with such large
> amount of email?

We return 550 during the SMTP conversation. If this was something from
a real human, they'd be contacting us another way.

Which is another benefit of RBLs -- with some MTAs, it's easier to do
the checking as part of the SMTP conversation, so sane error can be
returned to humans if indeed they were the original senders.

In my setups, I do run spamassassin immediately after DATA, so
legitimate senders should still get a sane error. We run it fairly
conservatively, and have only had one person fall afoul of this since we
deployed it a year or so ago.

-- John

[John Goerzen]
> We return 550 during the SMTP conversation. If this was something
> from a real human, they'd be contacting us another way.

Right. I believe this way of measuring false positives is inaccurate,
as I know some users will just curse and conclude that if the system
refuse to accept the email, they will just ignore the user behind that
email and leave those not getting the information in the email in
ignorant bliss. Several times I have tried to submit a patch to a
software proejct only to have the email rejected. In those cases, I
just leave it at that, because I do not want to spend more time on
people rejecting valid email.

> Which is another benefit of RBLs -- with some MTAs, it's easier to
> do the checking as part of the SMTP conversation, so sane error can
> be returned to humans if indeed they were the original senders.

Here at work they use the blacklists to decide the speed of the SMTP
responses. Hosts listed in blacklists get very slow response, while
others receive immediate response. A lot of spammers do not wait 30
seconds to 5 minutes for a response and just disconnect, while most
(all?) real mail servers have more patience. :)

Friendly,
--
Petter Reinholdtsen

Previously John Goerzen wrote:
> We return 550 during the SMTP conversation. If this was something from
> a real human, they'd be contacting us another way.

Personally I just give up on the person/organisation rejecting my mail.

Wichert.

--
Wichert Akkerman <wichert(at)wiggy(dot)net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.

Petter Reinholdtsen <pere(at)hungry(dot)com> wrote:
> [John Goerzen]
> > We return 550 during the SMTP conversation. If this was something
> > from a real human, they'd be contacting us another way.
>
> Right. I believe this way of measuring false positives is inaccurate,
> as I know some users will just curse and conclude that if the system
> refuse to accept the email, they will just ignore the user [...]

Where "some users" includes SPI Deputy Treasurer Branden Robinson who
used to note it on http://deadbeast.net/~branden/homepage/mailblock.html
which says "I find blacklisting based on originating IP, when that IP is
not a known source of spam, to be unethical and discourteous."
rfc-ignorant, dsbl and securitysage look like they can list non-sources,
but this is a more general ethic, as spotting false positives is hard.

> Several times I have tried to submit a patch to a
> software proejct only to have the email rejected. In those cases, I
> just leave it at that, because I do not want to spend more time on
> people rejecting valid email.

Yes, I've had that happen and it sucks. More when I was with Wanadoo or
Pipex - phonecoop is smaller and is less likely to have an unnoticed
listing for long, but it still sometimes happens. I published the
blocked patches, but some of them (better blind support for Mailman, for
example) haven't been applied AFAIK.

> Here at work they use the blacklists to decide the speed of the SMTP
> responses. [...]

That's clever. I do something similar on trackbacks and comments and it
mostly stops them after the first response. Is there a guide to setting
that up for mailservers?

Best wishes,
--
MJ Ray - see/vidu http://mjr.towers.org.uk/email.html
Somerset, England. Work/Laborejo: http://www.ttllp.co.uk/
IRC/Jabber/SIP: on request/peteble

On 10817 March 1977, MJ Ray wrote:

>> > We return 550 during the SMTP conversation. If this was something
>> > from a real human, they'd be contacting us another way.
>> Right. I believe this way of measuring false positives is inaccurate,
>> as I know some users will just curse and conclude that if the system
>> refuse to accept the email, they will just ignore the user [...]
> Where "some users" includes SPI Deputy Treasurer Branden Robinson who
> used to note it on
> http://deadbeast.net/~branden/homepage/mailblock.htmlwhich says "I
> find blacklisting based on originating IP, when that IP is not a
> known source of spam, to be unethical and discourteous."
> rfc-ignorant, dsbl and securitysage look like they can list non-sources,
> but this is a more general ethic, as spotting false positives is hard.

IIRC he had problems with idiots blocking IPs that are assigned to
DialUp users. SPI doesnt do that crap.

--
bye Joerg
<liw> I'm a blabbermouth

On Tue, Oct 24, 2006 at 07:05:46PM +0200, Wichert Akkerman wrote:
> > We return 550 during the SMTP conversation. If this was something from
> > a real human, they'd be contacting us another way.
>
> Personally I just give up on the person/organisation rejecting my mail.

Imagine if spammers behaved that way... :)

--
2. That which causes joy or happiness.