For Upcoming PF Tutorials, We Welcome Your Questions

© 2025 Peter N. M. Hansteen

A good tutorial should sound to passersby much like an intense but amicable discussion between colleagues.

In a little over a month, I'll be heading out to Ottawa to attend BSDCan 2025, to help run the conference and to give this year's Network Management with the OpenBSD Packet Filter Toolset tutorial, with my co-presenters Max Stucchi and Tom Smyth.

I've been a regular at BSDCan since 2006, attending every year since except 2008 -- I wanted to go that year too, but other business (actually the business of getting out of a company I'd helped build) kept blocking my preparations even though I had a fresh book out with the first edition of The Book of PF published late 2007. The most recent edition, the third, is available from here and from other good book sources.

Note: This piece is also available without trackers but classic formatting only here.

But I've kept coming back after that, and I've almost always given the PF tutorial at BSDCan, this year again there is a PF tutorial, Network Management with the OpenBSD Packet Filter Toolset, which is likely to follow the format from the previous years.

The format of the session is a sequence of short, intensive lectures interspersed with lab excercises. You can get a reasonable idea of what to expect by looking at the slides from the last sesssion. It can be worth noting that future updated versions of the slides will be available in the same location.

But we're still weeks away from the session, and we are in the period of time when we are making revisions and adjustments. So this is a good time to tell us what topics you would like to see us cover in the next session.

If you have one or more questions or suggestions, please do send them on to us at questions@pftutorial.net, preferably after taking a peek at the slides from the last sesssion.

We will consider each carefully, and if your question or suggestion turns into a new or changed topic, we will give credit where credit is due.

We are looking forward to seeing you in Ottawa in June!

Check out the BSDCan 2025 site for more information on the conference.

If you are unable to attend BSDCan, all is not lost: the EuroBSDcon 2025 conference is still accepting submissions for papers and tutorials, so if you have an interesting BSD-related topic you want the world to know about, your submissions will be welcome at the EuroBSDcon submissions system, where the deadline is 2025-06-21, or June 21st, 2025 (full disclosure: I'm on the program committee). This year's conference is set in beautiful Zagreb, Croatia in late September.

---

You might also enjoy

A Few of My Favorite Things About The OpenBSD Packet Filter Tools (2022)
The Hail Mary Cloud And The Lessons Learned (2013 - 2024)
What every IT person needs to know about OpenBSD (2021)

BSDCan Tutorials: Please Help Me Improve Your Experience

A good tutorial should sound to passersby much like an intense but amicable discussion between colleagues.

In a little over a month, I'll be heading out to Ottawa to attend BSDCan 2014. I've been a regular at BSDCan since 2006, attending every year since except 2008 -- I wanted to go that year too, but other business (actually the business of getting out of a company I'd helped build) kept blocking my preparations even though I had a fresh book out with the first edition of The Book of PF published late 2007.

Note: This piece is also available without trackers but classic formatting only here.

But I've kept coming back after that, and I've almost always given the PF tutorial at BSDCan, this year I'm branching out a bit to give two separate sessions:

Note: While this piece refers to past events, some points may apply to upcoming events too.

Building the Network you need with PF, the OpenBSD Packet filter

and

Transitioning to OpenBSD 5.5

Both sessions have been allocated 3 hour slots, and they also share another characteristic: I've invited my attendees to send me an email about what they're interested in learing during the tutorial. The main reason I do that is that I want to improve the experience for you, my prospective tutorial attendee.

Let me give you a bit of background. I've been giving the PF tutorial in various forms quite a few times over the years, and at one point I'd accumulated enough useful PF material that writing a book about PF seemed to be a natural next step. There was always some material that did not quite fit the book format, but a lot stayed at least for a while in my tutorial slides.

Over time I ran into a bit of trouble with the fact that BSDCan tutorials are always 'half day' or 3 hours. My collection of slides and notes have tended to expand over time, partly as a function of more experience, and partly due to the sad fact that the other BSDs have been slow to adopt any post-OpenBSD 4.5 syntax changes and other innovations. At some conferences and events I've done the PF session as a sometimes bit overfull full day event, depending on the number of questions and amount of other interaction.

But this time around it's three hours only, and I think that's quite an opportunity to improve the experience. I have more than enough material, but I've found that I usually know next to nothing about the people who will attend the sessions, and people's backgrounds vary enough that it's sometimes hard to find common ground or even pick out at short notice which parts of the material actually fits the group. I've had groups where some attendees had barely used any BSD at all along with OpenBSD developers who committed updates to man pages during the session in response to my slides and remarks, and most levels in between.

So with a strict limit on time, I would very much like to tailor the event specifically to the people who will be attending and who have something of an idea of what they want to learn. So please send me that email (to tutorial@bsdly.net), and I probably will end up updating the dense mass of slides anyway, and after the session they will be put in the usual place for browsing at your leisure.

If the format works, it's likely I'll try the short and tailored approach again at future events.

But there's more. As it says in the Transitioning to OpenBSD 5.5 tutorial description, OpenBSD has been the source of a number of BSD innovations over the years, and OpenBSD 5.5 has several noteworty improvements: time_t is now a 64-bit value so time will not wrap anytime soon, we have a new traffic shaping system wrapped into PF and a clear path to replacing the once-experimental but now aging ALTQ, signify(1)-signed install sets and packages, and quite a few more bits. The relase page is filling out nicely at the moment.

It's likely that those changes alone cold be made to fill a 3 hour tutorial slot nicely, but once again I would very much like to shape the session to fit the needs of the people who are planning to attend, so it's likely that more general what to look out for when switching to OpenBSD style material will be useful too. And if you haven't already, your experience will be much improved if you prepare a bit. The OpenBSD FAQ and the website in general is a valuable resource, and Michael W. Lucas' Absolute OpenBSD, 2nd edition is a very good source of information.

This year's BSDCan will be the first time I do the Transition to OpenBSD N.m session (unless I do get a rehearsal run organized with some locals), but it's likely I'll try again at later events for whatever is the just released or soon to be released OpenBSD version. Things are shaping up nicely for OpenBSD 5.6 at the moment, but the details of that future release will be out of scope for the Ottawa session. So please do send me an email (to transition@bsdly.net) if you plan to attend the session, and I will do my best to tailor the tutorial to your needs.

For both sessions, my ambition is to have the tutorial sound like an intense, but amicable discussion among colleagues. I look forward to seeing you in Ottawa.

Update 29 April 2014: A few people have asked, and I answer: Even if you're not able to attend the BSDCan session, you're of course welcome to send me questions to indicate what you would like to see covered in the tutorials and in the slides I'll publish afterwards. I won't give a firm promise to cover every question, but I'm happy to hear from you.

---

In other things, the manuscript for the third edition; of The Book of PF -- which main reason to exist is the new traffic shaping system -- is complete, going through the various editing steps and will be available at a yet to be determined date in 2014. I will be updating here and through twitter, G+ and other channels once more detailed information is available.

If you are unable to attend BSDCan, all is not lost: the EuroBSDCon 2014 conference is still accepting submissions for papers and tutorials, so if you have an interesting BSD-related topic you want the world to know about, please drop us a line (or even better a title, abstract and short biographical description) at submission@eurobsdcon.org (full disclosure: I'm on the program committee). This year's conference is set in beautiful Sofia, Bulgaria in late September.

Anticipating the Post-ALTQ World

A peek into exciting new features in the upcoming OpenBSD 5.0 release and beyond, plus how to teach our favorite operating system better?

One of the more exciting pieces of news to come out of the Edmonton OpenBSD hackathon this week was that the face of OpenBSD traffic shaping is about to change.

This change is by no means the only news to come out of Edmonton (read source-changes@, either by subscribing or via public archives such as marc.info and, if you're at all BSD geekish, feel your excitement levels rise), but the new prio keyword that was added to PF grammar and announced via a message to the tech@ mailing list titled new small, fast, always on priority queueing is both a major new feature and a prime example of how the OpenBSD project works, avoiding sudden "revolutionary" steps in favor of well planned evolutionary steps. The stepwise approach ensures, among other things, that changes are properly tested, and is visible in the fact that the OpenBSD source tree is as close as doesn't matter to always being in a buildable state, even during hackathons.

ALTQ, the ALTernate Queueing subsystem, was integrated into the PF codebase for OpenBSD version 3.3 (mainly by the efforts of the very same Henning Brauer who posted the message linked in the previous paragraph), and from that release onward, managing your filtering and your traffic shaping via your pf.conf file has been a highly appreciated feature of OpenBSD and other systems that have adopted the PF networking toolset.

Although much appreciated by its users everywhere, the integration was never quite an easy fit and the developers have been privately discussing how the queuing should be rewritten to avoid the separateness of ALTQ compared to the rest of the PF features.

There have been other concerns too. Not all of the features outlined in Cho's original USENIX paper were ever fully implemented, and the code had begun showing its age in several respects. By amazing coincidence another mailing list thread appeared in the same week as the new queueing system was announced, pointing out that ALTQ's total bandwidth parameter is a 32-bit value, meaning in practical terms that the maximum bandwidth ALTQ will be able to manage is in the 4Gbit range. Just changing the relevant variables to a wider type apparently breaks the code in other interesting ways (if you're adventurous, you could try playing with FreeBSD developer Ermal Luci's patch (available among other places here) and see what happens). The results could be hackishly interesting, but with the new queueing system set for gradual introduction, this may not be the optimal time to submit ALTQ patches.

Once again, the 32 bit value is a sign of the code's age. At the time ALTQ was originally written, a 32-bit value for bandwidth, and by extension an absolute cap at four gigabits per second, was a no less reasonable choice than the choice of 32 bits as the length of IP addresses some years earlier. And while we are still in the ALTQ world, the obvious workaround in settings where you have bandwidth that comes in increments of ten or forty gigabit is to do your filtering and traffic shaping at network locations where bandwidth is a bit more scarce. In the post-ALTQ world, things may become significantly different.

The early outline of the new world comes in the form of in-ruleset traffic classification, at the face of it not that different from ALTQ when seen from a user perspective. In the upcoming OpenBSD 5.0 release onwards, you will be able to skip the familiar ALTQ queue definitions, but still do traffic classification like this:

pass in proto tcp to port ssh prio 6
meaning that incoming ssh traffic will pass with a relatively high priority (the range is 0 through 7), hopefully making it through earlier than the rest.

The new scheme even lets you duplicate the old speedup trick of setting aside a separate subqueue for ACKs and other lowdelay packets, like so:

pass in proto tcp to port ssh prio (5, 6)

This has yet to reach downloadable binary snapshots, but if you have the required bits in place, you can get an early start by building your own OpenBSD-current. See the relevant pieces of The FAQ, supplemented by man release.

It is important to note that in the upcoming release, priority rules will coexist with the existing ALTQ infrastructure. Over time, however, the plan is to replace ALTQ with a largely rewritten system where the internal workings of the HFSC and CBQ queues, for example, are not all that different. We are also likely to see the total bandwidth definition for an interface move out ouf your pf.conf and re-emerge as an ifconfig option.

Once a traffic shaping infrastructure that's functionally equivalent to ALTQ (or it is hoped, superior in all ways including performance) is in place, ALTQ will eventually be removed. In the meantime, we have already seen commits that set default priority for certain traffic types, and reading source-changes@ along with testing snapshots will continue to be a fun and exciting experience in the weeks and months ahead.

Over time, it might even become necessary to do more book work, if so you will hear about it first here.

Whatever happens on the book front, I'll more than likely get back to those new features in more detail in upcoming PF tutorial sessions (See here, here and here for announcements about the ones scheduled so far), and I'm trying to figure out a way to improve the quality of the tutorial experiences.

The lecture plus demonstrations format of the tutorial sessions has so far allowed for very limited interactivity with little or no hands-on experience for attendees during the sessions themselves. I would be very happy to hear ideas for and input on practical implementation of changes that would improve the experience. One possibility is setting up a set of virtual labs, hosted somewhere reachable from the conference locations. Suggestions are welcome via email or the comments field.

Update 2011-07-10: The prio code is in snapshots dated July 10, 2011 or newer. If you're upgrading from a previous version, the installer will now also offer to run sysmerge(8) and it will even offer to upgrade any non-free firmware it finds installed on your system.

Update 2011-09-14: Chris Cappucio wrote in, adding some more useful detail to ALTQ's history. Chris writes:

I did the initial altq port to OpenBSD and Kenjiro Cho (author of ALTQ) was invited to the next hackathon (I believe it was c2k2) along with myself to integrate it into the tree. I declined to participate for personal reasons but Kenjiro did turn my patch into an in-tree implementation, one or two years later the PF guys replaced the ALTQ packet classifier and configuration utility with PF itself. Now they are replacing the ALTQ scheduler framework completely with a simpler framework that still uses the PF classifier and configurator :)

Update 2015-04-02: The new queueing subsystem finally appeared in OpenBSD 5.5, and was the main motivation for revising The Book of PF into its 2014 third edition. If you follow the book links in this article, you will find the edition that describes both the new queueing system and the legacy ALTQ system.

How Do We Appropriately Celebrate The Arrival Of The 100,000th PF Tutorial Visitor?

A nice surprise may be in line for a new visitor, and you (yes, you) can help me pick the surprise.

In late 2004, I started working on the text for a user group lecture for the BLUG meeting scheduled for the the following January.

The original manuscript was in Norwegian, but after a rather successful and surprisingly well attended user group meeting, I wrote up an English version and posted both online. With some encouragement from Greg Lehey (I'd participated in the group of volunteer reviewers for his third edition of The Complete FreeBSD), I submitted a proposal to give a half day tutorial for the 2005 AUUG conference in Sidney.

The proposal was accepted, as were several of the followup submissions to other conferences, and via a sequence of conferences and some private sessions, the document kept developing. In early 2007 I started working on turning the manuscript into a usable book. As regular readers will be aware, the much revised second edition was completed during the second half of 2010, and even that version has recently been subjected to its first update thanks to the ongoing development of the OpenBSD operating system.

The original tutorial has kept attracting a relatively steady stream of new visitors from all over the world, even though I have not added any new material to the document since I started working on the book version. New material will, rather, find its way into slides for the next session (such as the most recent one at BSDCan 2011), or will be put in the queue for possible upcoming book material.

During periods when I have had little visible output to offer, it has been interesting to see that the documents attract visitors and the occasional comment or suggestion for improvement. Then a little while back, I realized that in a not too distant future, the number of unique host names or IP addresses that have visited the tutorial tree will roll past a hundred thousand (100,000).

That particular number is possibly only significant to me, I keep the count of unique hosts accessing mainly to get an idea how many people have looked at my work. The raw number of page hits for the same location (we don't have any numbers for the early days when it was hosted at a now-defunct ISP) is fairly close to one and a half million, but I feel that number is a rather pointless statistic.

But when visitor number one hundred thousand arrives, how should we celebrate? I'm inclined to try to identify and contact the lucky visitor and offer a prize of sorts, but I have not quite made up my mind what and how. I'll welcome suggestions sent to via email (to peter@bsdly.net) with a recognizable subject.

It is worth mentioning that neither the tutorial nor this blog directly generates any revenue for me. I did for a short time have Google-supplied ads running on both sites, but for reasons that have never been quite clear to me, Google chose to terminate my AdSense account a few days before my second USD 100 transfer was due.

I will not mindlessly paste from HOWTOs

Even with proper discouragement, mindless pasting is rampant, it seems

It had to happen sooner or later.

My incoming mail this morning had one item about what I thought was a fairly trivial misconfiguration, and I answered it like this

From: peter@bsdly.net (Peter N. M. Hansteen)
Subject: Re: interesting-traffic
To: Name Withheld <Name.Withheld@gmail.com>
Cc: peter@bsdly.net
Date: Sun, 30 Jan 2011 12:44:35 +0100

Name Withheld <Name.Withheld@gmail.com> writes:

> how should i handle the 'intersting-traffic macro not defined' error
> in pf.conf on obsd 4.8 reboot syntax error starting pf?

either define the macro (remove a # comment perhaps) or remove any
references to it.  Have you been pasting from a partial example
floating around the web perhaps?

- P

Then a few sips of coffee later, it dawned on me: the macro interstring-traffic is more than likely one I made up for the bridge example in the short (and now rarely updated) version of my PF tutorial document. (I added the strongly worded note there as a reaction to this incident).

So it's at least partly my fault. I put an incomplete example out there, hoping that whoever stumbled upon the material would grasp the context and fill in any needed details. The important bits are all there, but when pasted into a config without checking, the result will be just as Name Withheld experienced.

But then I can't really take the full blame: Had he bothered to read the rest of the document or even the book that's a further development, he would have seen this admonition which comes out even more clearly in the slides version. If for some reason the links are inoperative, here it is:

The Pledge of the Network Admin

This is my network. 

It is mine 
or technically my employer's, 
it is my responsibility 
and I care for it with all my heart

there are many other networks a lot like mine,

but none are just like it.

I solemnly swear 

that I will not mindlessly paste from HOWTOs.

I actually recite that at the very beginning of all my tutorial sessions, and while of course it's sometimes accompanied by giggles, the point remains: there is no substitute for actually understanding your configuration. Testing (if nothing else, a quick sudo pfctl -vnf /etc/pf.conf and reading the output before rebooting) would have helped enormously too.

---

For those hungry for fresh PF tutorials, I'll jump the gun and announce that there will be one by yours truly at AsiaBSDCon 2011, final schedule to appear on that URL shortly. A few other events are in the works too, more details here and at the PF tutorial page when details are settled.

The Book of PF, 2nd ed: It's Here!

Yes, it's that time of the year again -- we missed both Halloween and the OpenBSD 4.8 release, but hot on the heels of both, here it is:

The Book of PF, 2nd Edition is here, a box of author's copies turned up here just after lunchtime, and were taken well care of by Nora and Birthe.

This means, of course, that those of you who preordered will be receiving your copies shortly (mod the usual factors eloquently described by Michael Lucas here, the printer in my case is in Louiseville, Quebec), those who have reason to expect copies from my hoard here can rest assured that I'm taking them to the post office right after this. There's an illegible scrawl on some early pages, sorry 'bout that.

Better bookstores online and elsewhere will have it, or you could make it part of a bundle by ordering from the OpenBSD orders page. You will be going there for your six monthly fix anyway, won't you?

Upcoming events: The plans are not fixed yet, but you should expect me to turn up at BSD-themed events over the next few months. Look for announcements here, tweeted, or via the usual mailing lists.

NOTE: This article refers to the now outdated second edition, which has been superseded by The Book of PF, 3rd Edition, which covers changes up to and including OpenBSD 5.6. For the purpose of learning network technology in general and PF in particular, the significantly updated third edition is a better choice than the second edition. Also see the October 25, 2015 post about the arrival of my third edition author copies.

Into a new year, slowly pounding the gates

The distributed but clearly coordinated bruteforcers are still at it. How long until they reach the end of the alphabet? And why are they staying away from my OpenBSD machines? Are we seeing the contours of a controlling intelligence?

As large parts of the Western world prepares for the holidays, the swarm of little robots that started trying to pry open the doors to my machines some weeks back are still at it. As far as we can tell, the coordinated attempts started some time in early November or perhaps late October (we don't keep logs around for long enough to be sure), with an alphabetic progression that has now progressed to somewere into the os. The complete listing from the time I started noticing up to the time I started writing this column can be found here.

I've written about this before, and in fact one of those columns was slashdotted, a pleasant surprise to me and a cause of some excitement among my colleagues at FreeCode.

After writing that article, I did some further research and found out that a precursor to what we are seeing now was observed as early as May 2008, as described in an Ars Technica article published at the time. That article also reveals, via Linuxtoday, that yours truly was among the many who failed to understand the problem, at least for a while. Then again, maybe actual log excerpts would have helped.

The problem, such as it is, seems to be that a somebody who herds a botnet has decided that the laws of big numbers favors those who keep trying for long enough. User names and passwords are generally far enough from random that if you are allowed to go on for long enough, you will sooner or later manage to guess a correct combination of username and password and get access to a machine somewhere.

Sysadmins have been seeing bruteforce attacks for years. The traditional brute force attack would be a rapid succession of login attempts from one host, and usable countermeasures were devised in short order. My favorite of course involves PF, and the description of how to thwart traditional bruteforcers is one of the more popular pages in my PF tutorial.

The distributed, slow bruteforcers are different. For one, the login attempts from each host out in the cloud are spaced far enough apart in time that intrusion attmpt detectors will not trigger. Next, it takes a keen eye to spot the common thread in the attempts spaced up to a number of minutes apart: a monotonously alphabetic progression of user names, with attempts coming in from different hosts. Some number of attemtps at a specific user name, before the cloud moves on the next one, in alphabetic order.

During the period we have been observing the slow brute activity, a total of 695 hosts have been involved. A total of 665 hosts made unsuccessful attempts at authenticating at the hosts we are observing during November, while the number for December so far is 346. The typical number of attempts per user name has decreased, too, from a typical ten do fifteen during the early days down to between one and four during the last couple of weeks.

I thought at first that the decrease in activity was just an indicator that compromised hosts were getting cleaned up, but my colleague Egil Möller was the first to suggest that since we know the attempts are coordinated, it is not too far fetched to assume that the controlling system measures the rates of success for each of the chosen targets and allocates resources accordingly.

If Egil's assumption is right, we are seeing the bad guys adapting. My systems do not run any services they do not need to, and apparently all attempts at gaining access have been futile so far. So, the controlling system shifts resources to elsewhere, even if the access attempts do not stop entirely. Come to think of it, I'm not seeing any attempts at all on my OpenBSD systems, so it is possible to speculate that whoever is behind this phenomenon has decided that OpenBSD systems are hardened enough to begin with and usually run by compentent paranoids as to be useless as targets. That would be a comforting thought at the end of a long and sometimes trying year.

Speaking of the new year, look for exciting announcements coming from FreeCode. We're working on some cool things. And with a bit of luck, I might run into you at one conference or the other during the coming year.

Happy holidays to everyone.


Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

Logfiles in the buff

Search engine optimization, deflowered.

Logs are important. Depending on the specific kind of log, the data may shape lives and generate fortunes (how many times were those ads displayed, your clickthrough rate), reveal suspicious behavior and trigger actions (such as shutting the door to that bruteforcer) or provide sysadmins such as yours truly a general idea of what works and not or anything inbetween.

If you're a sysadmin, log data or log data derivatives such as a monitoring tool's graphical status display is more likely than not an important underlying factor to determine how you spend your day.

Then of course most of the material for these columns comes from log files, too. Depending on the specific log file, I tend to either just peek at the data my monitoring scripts offer me or do some manual greping for any patterns that interest me.

One such pattern matches the filename for my resume. I put that online for job hunting purposes, and now that I'm basically a gun for hire, it's slightly interesting to see any activity involving that file.

So at semi-random intervals, I check the apache log for references to my resume. Today, the grepery turned up this nugget

92.48.107.33 - - [27/Aug/2008:04:41:12 +0200] "GET /%7Epeter/PNMH-cv.html HTTP/1.0" 200 12318 "http://afmfokuv.fcpages.com/hot-anime-lesbians.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"

and I count myself lucky that I had thoroughly swallowed my last mouthful of coffee before reading that.

In the Era of PageRank, the Age Search Engine Optimization Consultant, Season of the Clickthrough rage, I suppose we should not be entirely surprised to see such things. Just what the two documents have in common, perhaps other than targeting a very specific market, is left as an excercise for the terminally curious. I would advise some caution in choice of browser and operating system if your research takes you to the referring URL. One of the lessons of the day is, it doesn't always take a spamd log to crack you up.

PF tutorial in London, November 26
In other news, the UKUUG are hosting a full day PF tutorial featuring yours truly in London on November 26th, 2008. See the UKUUG web site for details. OpenCON is the following weekend in Venice, and I hope to make it there too.

The Name and Shame Robot
Last week the Norwegian edition of Computerworld published an article about the Name and Shame Robot, unfortunately in the paper edition only (yes, I've got an English article in process too). The article did spur some nameandshame.html traffic from unexpected places, but no offers of cooperation or spamd synchronization so far. In the meantime, I'm running into odd cron behavior differences when trying to run the generator script I wrote on my OpenBSD machines on a few FreeBSD hosts. More than likely there is a lesson to be learned there too.

More than 40,000 served

Today's blog comes to you from sunny Aalborg in northern Denmark, where our Danish friends had the good sense to put together a one-day conference. Go to the web site at http://www.foss-aalborg.dk/ for details of the programme, I certainly hope the organizers will start a tradition and put on another conference soon.

For my own part, the PF tutorial (the free predecessor to the book), saw its confirmed unique visitor number forty thousand today, apparently a user somewhere in Ukraine:

$ ./mystats.sh
Wed Jun 4 14:57:26 CEST 2008
Total PF tutorial visitors : 40000

I promise I won't bother you with updates of the number of visitors again until we reach fifty thousand. I'll do my very best to have produced some other interesting material before then.

Riga, here we come, OpenBSD 4.3 on the horizon

On Wednesday, Riga is the place to be, if attending a PF tutorial session for free is how you want to spend a day. You may have missed the announcement (and the update), but no matter - on Wednesday, February 20th I will be giving a full day tutorial in Riga, Latvia. bsdly.net has a Riga event page which is the space to watch for updates about the event.

A few things about the event is settled, though: The content will have quite a bit in common with the book, and I will be bringing a some copies of in case you need to pick up one (and promo postcards if you just want a souvenir). And time allowing, we will be previewing some of the things we can expect to see in the upcoming OpenBSD 4.3 release.

OpenBSD 4.3-RELEASE is likely to be cut in a few weeks' time (with a likely release date May 1st), so if you want to get an idea what it is going to look like or even want to help track down and squash any remaining bugs, now is the time to grab a snapshot and start putting it through the paces. If you're a little bit like me and can spare some capacity for testing, you will enjoy the process.

The next chance to catch a PF tutorial by me will be at the UKUUG Spring 2008 conference (not a free as in beer event), unless you arrange something yourself. This year's events look like they'll be a lot more ad hoc than earlier, so there may be other events announced later. Keep an eye on the OpenBSD Events page and any updates here.

Also keep an eye out for the upcoming premiere issue of BSD magazine. It promises to have about 60 pages of BSD related material in the first issue. I have an article in there that I hope you will enjoy.

See you in Riga or some other time,

A year ends; what to do next?

It's the end of a year already. The end of the year is among other things the traditional time for tallying up totals to see what the year brought and looking forward to the fresh year ahead. Now at this point the inner geek in me and probably you too rebels with Why should any arbitrarily chosen point in time be assigned that much significance, huh?, but let's face it, it's one convention we will just have to live with.

The past year included a number of events, some entirely expected like the formal beginning of the end of the corporation once known as Caldera (yes, I know, it's not quite over yet, and I've written about that earlier), some rather surprising like the recent EU brokered patents and specifications deal which apparently means that the Samba team and other interested parties will not only be given access to usable protocol specs, they will even be furnished with a list of what Microsoft believes to be their relevant patents. That at least puts a serious dent in that corporation's patent FUD capability.

Any pundit in the Microsoft/Linux/FOSS "watcher" crowd who left that one out of their year end summary pieces should consider themselves cautioned: You were not paying attention to what could be this year's most important single piece of news in our field.

The general picture of the IT field is rather one of vast crowds of users who simply want to get on with their lives. The typical user is weary of the seventeen and a half times a week ritual Microsoft malware scare, and doesn't really see any benefit in getting a new computer with Vista to slow it down, now that they've finally weeded out or gotten used to all the annoyances of Windows XP and the background noise of unwanted popups and spam.

Rather more depressingly for us in the FOSS field, the typical user wants to just get on with his or her life and is weary, too, of the constantly overhyped "solutions" IT types are peddling. Faced with < insert your favorite product selling point here > , the stock answer now is, "gimme a break, that's what the last one said too".

This goes for almost any selling point, including vastly improved security along any measurable axis, efficient spam killing (including avoidance techniques like greylisting), the lightweight while useable desktop, and during the past year Microsoft even made a credible attempt at taking "open standards" prisoner. There is clearly a lot of work to be done, and we need to find ways to do that work better and present it in ways that actually add to FOSS people's credibility.

That includes, in my view, finding better ways to handle the periodic squabbles over licenses such as the GPL vs BSD shouting matches. It is likely that I will return to that topic in a future column, if and when I find the time to write it properly.

In my own little corner of the world, the publication of The Book of PF, marked here by the arrival of the author copies, marked the end of a long process that consumed rather more time and resources than I had anticipated. Before those copies arrived I had some copies made for OpenCon which were auctioned off for amazing sums that were subsequently donated to the OpenBSD project (see undeadly.org for details). Even though Amazon.com now lists the book as due for release January 11th, I have confirmation that No Starch shipped all preorders before they closed for the holidays, and I know at least one correspondent who got a message from the UK arm of Amazon that his copy was on its way. I'm interested in hearing from you about the book, of course, even reports that it has arrived safely in your mailbox.

Now other opportunities beckon, and I promise that in the coming year I will be writing about developments, confidentiality agreements allowing. If there is anything specific you want me to write about, please let me know.

I give you all my best wishes for the new year.

PS I almost neglected to mention that the PF tutorial (the forerunner of the Book of PF) saw its visitor (unique IP address or host name) number 27,000 for the period we have log data for on December 24th.

Update 2015-04-02: The Book of PF is now in its third edition, and the link in this article has been changed to point to the more recent edition.

I Must Be Living in a Parallel Universe, Then

It's Sunday morning, and I'm having my morning coffee while getting ready for a long session of editing my OpenCON presentation. By working on adapting the presentation tailored to the tutorial I've been rediscovering just how much work went into making the book, so a long Sunday session is needed, if not more.

Then courtesy of Groklaw's news picks comes the USA today piece called Despite filters, tidal wave of spam bears down on e-mailers.

A tidal wave of spam, no less. Well, we're seeing a lot of attempts at sending, like the sequence here (text link, formatting it would take too long) that I captured from the xterm running a tail -f on my spamd log a little while back. That sequence tells me, for one thing, that the naive spambot thinks my spamd looks like an open relay.

The other interesting thing about the sequence there is the pattern you can see in the From: addresses. It may have dawned on some of the spammers that generating random addresses in other people's domains might end up poisoning their own well, so they started introducing patterns to be able to weed out their own made up addresses from their lists. I take that as a confirmation that our harvesting and republishing efforts here and elsewhere have been working rather well.

Here the method seems to be that they take the victim domain name, prepend "dw" and append "m" to make up the local part and then append the domain, so starting from sia.com we get dwsiam@sia.com.

There is one other common variation on that theme, where the prepend string is "lin" and the append string is "met", producing addresses like linhrimet@hri.de, used just a few minutes ago to try to spam malseeinvmk@bsdly.net from the apparently Polish adress 89.228.40.80. This is of course very interesting, as is the fact that right now about two and a half thousand machines are in my spamd-greytrap list . That's where they end up, making no waves at all.

On the subject of patterns, earlier this month the address capitalgain02@gmail.com started appearing frequently enough that it caught my attention in my greylist dumps and log files.

The earliest contact as far as I can see was at Nov 10 14:30:57, trying to spam wkzp0jq0n6.fsf@datadok.no from 193.252.22.241 (apparently a France Telecom customer). The last attempt seems to have been ten days later, at Nov 20 15:20:31, from the Swedish machine 217.10.96.36.

My logs show me that during that period 6531 attempts had been made to deliver mail from capitalgain02@gmail.com via bsdly.net, from 35 different IP addresses, to 131 different recipients in our domains. Those recipients included three deliverable addresses, mine or aliases I receive mail for. None of those attempts actually succeeded, of course. With a little more time on my hands I'm sure I could have made a good regular expression to calculate to the second how much time those spam senders wasted here, too.

So where's the tidal wave? Back when PDF spam was the new horror, it actually took three weeks for one to reach me, and then only via an alias on a machine I really don't have much control over anymore. The number of spam sending machines does seem to be increasing, though.

Bob Beck's uatraps list is a good indicator, and the tendency is clear from the graph in my malware paper. The number did dip just below 100,000 addresses earlier this month, and it now seems to have stabilized in the 110,000 to 120,000 range.

From my perspective, it looks like a reasonably configured spamd is really all we need to observe the tidal wave at a safe distance and have fun all the while.

It's almost like living in a parallel universe.

Of Course, It Had To Be A Webshield

In an earlier blog post, I mentioned that I would buy a round of drinks the first time I saw an attempt to deliver a message with both the From: and To: addresses already on my spammer baiting list.

In fact it happened very soon afterwards, and as luck, misfortune or just plain old incompetence would have it, that message apparently came from a WebShield appliance not too far from here:

Oct 17 23:03:52 skapet spamd[20795]: 194.54.96.18: connected (6/4)
Oct 17 23:04:03 skapet spamd[20795]: (GREY) 194.54.96.18:
<capitulations7@datadok.no> -> <capitulations7@datadok.no>
Oct 17 23:04:03 skapet spamd[20795]: 194.54.96.18: disconnected
after 11 seconds.
Oct 17 23:19:21 skapet spamd[20795]: 194.54.96.18: connected (4/3)
Oct 17 23:19:32 skapet spamd[20795]: (GREY) 194.54.96.18:
<capitulations7@datadok.no> -> <capitulations7@datadok.no>
Oct 17 23:19:32 skapet spamd[20795]: 194.54.96.18: disconnected
after 11 seconds.
Oct 17 23:30:30 skapet spamd[20795]: 194.54.96.18: connected (4/4),
lists: spamd-greytrap
Oct 17 23:34:14 skapet spamd[20795]: (BLACK) 194.54.96.18:
<capitulations7@datadok.no> -> <capitulations7@datadok.no>
Oct 17 23:35:58 skapet spamd[20795]: 194.54.96.18: From:
Webshield.SMTP.V4.5.MR1a.Mail.Service@vs4.bgnett.no
Oct 17 23:35:58 skapet spamd[20795]: 194.54.96.18:
To: <capitulations7@datadok.no>
Oct 17 23:35:58 skapet spamd[20795]: 194.54.96.18:
Subject: Returned Mail: Error During Delivery
Oct 17 23:37:00 skapet spamd[20795]: 194.54.96.18:
disconnected after 390 seconds. lists: spamd-greytrap
Oct 17 23:57:18 skapet spamd[20795]: 194.54.96.18:
connected (6/6), lists: spamd-greytrap

I sent the operators at that site a polite message right away, pointing out the misconfiguration. Two weeks later I have seen no response other than the automatic acknowledgement, but it looks like the machine has managed to get itself automatically whitelisted in the meantime. So perhaps they found the button that actually does something.

Since my last blog post I have completed the book, and I expect the last bit of proofing to be done during the coming week. Then a few other necessary processes, and physical copies available for mid December if all goes well. With the cover in place, it looks like it's become attractive and popular over at amazon.com in its various categories. The BSD category there looks pretty No Starch dominated at the moment.

That can not be a bad thing. It's been a real pleasure working with the people at No Starch Press. If you think you want write a tech book, they should be on the list of publishers to contact with your proposal.

While all this was happening, the spammer baiting operation seems to have reached a critical mass of sorts. With roughly 7,200 addresses in the spamtrap list there are several hundred bait addresses for each real one in those domains taken together, so it's extremely unlikely that the spammers will ever get a chance to try delivery to a real address before they hit the tar pit. Over the last couple of weeks, my gateways have had anywhere between 2,500 and 4,000 hosts in the local spamd-greytrap, and anywhere from 0 to about 300 spambots pushing bytes into the tar pits at any time. It's fun to watch (some of the bots labor through the bait list from top to bottom), and the net effect is, well, we're not seeing much spam.

I think I've mentioned it before, but it bears repeating: To naive spammers and the tools they use, spamd looks like an open relay. Spamd never actually delivers any messages, but this


GREY|201.250.57.147|sofia|<vdaegkoxgk@bonana.com>|
<brad.james.anderson@jhg.com.au>|1193105605|1193127205|1193127205|1|0

says that whoever operates 201.250.57.147 (according to whois, likely located in or near Buenos Aires, Argentina), is unable to tell the difference between an open relay and spamd's 451 and subsequent "this is going to hurt you more than it hurts me" messages.

Another variation on that theme is what I think is some sort of amateurish relay testing, which typically produces anywhere from five hundred to a thousand greylist entries of the type


GREY|59.35.4.51|UATIM-F7E7949C7|<adgjnq@194.54.103.104>|
<ariel5268@yahoo.com.tw>|1193084672|1193113472|1193113472|2|0
GREY|59.35.4.51|UATIM-F7E7949C7|<xaehkn@rosalita.datadok.no>|
<ariel5268@yahoo.com.tw>|1193084675|1193113475|1193113475|2|0
GREY|59.35.4.51|UATIM-F7E7949C7|<qswyd@brutha.datadok.no>|
<ariel5268@yahoo.com.tw>|1193084691|1193113491|1193113491|2|0
GREY|59.35.4.51|UATIM-F7E7949C7|<nqtw@monalisa.datadok.no>|
<ariel5268@yahoo.com.tw>|1193084733|1193113533|1193113533|2|0

where the From parts are made up of host names and IP addresses in our local net, including in this case, the host name for one of our laser printers. Those floods have tended to swell the bait list a bit, even if I strip out the invalid @<IP address> ones.

Spamd makes the naive relay testers think we have a whole network of open relays, and we harvest the noise they generate to lead the spambots to the tarpit. That's pretty close to a hands-off spammer repellent for us, and a serious auto-LART for the spammers.

OpenCON is sneaking up on us in a month's time, and we're heading for Venice with a refreshed tutorial session. See you there!

PS - [non-IT PS coming up] Bergen's football (soccer) team SK Brann has just won the national league for the first time in 44 years. With one game to go before end of season they are so far ahead in points there is no way any other team will be able to catch up. The town is predictably going gaga over the event, and we joined the thousands at the central Festplassen square for the city sponsored celebration tonight. I'm surprised how many songs have been written about that team and how everybody around me seened to know every last word of the lyrics. Good fun, ending with fireworks.

Always a pleasure to be wasting your time, guv

This week has been a little unusual around the BSDly household. So far I've generally been doing my regular job in the daytime (with longish office hours), only working on the book evenings and weekends. That the arrangement would lead to "Exhaustion is my middle name" status was obvious to everyone except me, but I finally saw where it could be going. So for a little more than the past week I've been working on the book full time.

The state of perpetual exhaustion has had some not too happy consequences. Of course the general progress on the book suffered, but it also lead to me missing the monthly BLUG meeting in August. Of course much of that particular day I had spent persuading somebody not too bright that it indeed had to be a reconfiguration they said had never happend at their end which ended up breaking things at our end, and I was just too tired and missed what I assume was a well executed lecture on networking basics by Vegard Engen (of RFC1149 implementation fame).

This week with only one job I needed to tackle, I was there for an enjoyable one and a half hours of Bacula, well presented by Bård Aase (aka elzapp). Off to Henrik (the regular BLUG pub) for a few beers afterwards, and with Johan Riise volunteering to put together a 'Unix and time' lecture for next month, the BLUG calender seems to be in order after all, with Jill Walker doing the end of semester talk in November, on whatever interesting stuff she has been up to lately. Unfortunately it looks like the last Thursday of November is close enough to OpenCON that I'll likely miss Jill's session.

In the meantime, there are signs that the greytrapping and my bait list is working. Looking over the spamd logs today I found quite a few entries like these:

Sep 29 15:29:23 skapet spamd[20795]: (BLACK) 84.76.177.159: 
<royaleuromillion2007@yahoo.es> -> <211hgsreliart7@datadok.no>
Sep 29 15:29:32 skapet spamd[20795]: (BLACK) 84.76.177.159: 
<royaleuromillion2007@yahoo.es> -> <00b27f18@datadok.no>

which looks strikingly like the Spanish lottery scam spammers patiently and methodically working their way through my list of bait addresses, all the way from top to bottom, at roughly 3000 addresses it's going be a while. All I can say is, we are extremely pleased to be wasting your time, senor.

Also while the girls were off to the Raptus comics festival (an annual event, and one of the big things here in Bergen), I found enough trash backscatter to non-existent bsdly.net addresses that it's likely that the same weekend spambot operators who spewed their spam with @ehtrib.org and @skapet.datadok.no addresses earlier (both times at weekends) have now discovered bsdly.net and are doing their damnedest.

Why they prefer to generate a few hundred fake addresses and use them all in one go is beyond me. The other groups seem to generate only a handful of new addresses each every day, and for good measure at least one of them sort of reuse the generated addresses by using a forward and a reverse (such as in this morning's preserved greylist dumps, there was a potterv76@datadok.no as well as the reverse 67VRETTOP3@datadok.no). This lot just dumps all they have in one go, mainly contributing to swelling that file in my home directory with the totally unprintable file name which is the temporary storage before they go to into the traplist and on to the bait page.

Distractions of that kind from my main task is never entirely welcome, but with a larger influx of new addresses to be added to the bait list I made some small changes to make the maintenance of that page a bit more sane, rediscovering server-side includes and redirects along the
way. And the data I keep collecting may become the basis for other projects later.

Anyway, it is increasingly clear that the spammers are including the generated fake addresses in their "known good" lists. Consider the spambot at 210.111.190.216 (apparently in Korea), which insists on delivering to an address somebody generated in early July:

peter@skapet:~/www_sider$ grep  210.111.190.216 /var/log/spamd
Sep 29 15:58:07 skapet spamd[20795]: 210.111.190.216: 
connected (5/4)
Sep 29 15:58:21 skapet spamd[20795]: (GREY) 210.111.190.216: 
<jim.vance@presentsmadeeasy.com> -> 
<careersogt2083@datadok.no>
Sep 29 15:58:22 skapet spamd[20795]: 210.111.190.216: 
disconnected after 15 seconds.
Sep 29 15:58:35 skapet spamd[20795]: 210.111.190.216: 
onnected (4/3)
Sep 29 15:58:49 skapet spamd[20795]: (GREY) 210.111.190.216: 
<tbaker@groupecdb.com> -> 
lt;careersogt2083@datadok.no>
Sep 29 15:58:50 skapet spamd[20795]: 210.111.190.216: 
disconnected after 15 seconds.
Sep 29 15:59:03 skapet spamd[20795]: 210.111.190.216: 
connected (5/3)
Sep 29 15:59:17 skapet spamd[20795]: (GREY) 210.111.190.216: 
<wotan@4vsi.com> -> <careersogt2083@datadok.no>
Sep 29 15:59:18 skapet spamd[20795]: 210.111.190.216: 
disconnected after 15 seconds.
Sep 29 15:59:30 skapet spamd[20795]: 210.111.190.216: 
connected (6/5), lists: spamd-greytrap
Sep 29 16:03:14 skapet spamd[20795]: (BLACK) 210.111.190.216: 
<sylviacastleman@alltypecalligraphy.com> -> 
<careersogt2083@datadok.no>
Sep 29 16:04:59 skapet spamd[20795]: 210.111.190.216: 
From: "Marguerite Casey" <sylviacastleman@alltypecalligraphy.com>
Sep 29 16:04:59 skapet spamd[20795]: 210.111.190.216: 
To: <careersogt2083@datadok.no>
Sep 29 16:04:59 skapet spamd[20795]: 210.111.190.216: 
Subject: 100mg x 60 pills US $ 129.95 buy now
Sep 29 16:06:04 skapet spamd[20795]: 210.111.190.216: 
disconnected after 394 seconds. lists: spamd-greytrap

I have no real opinion on the validity of the From: addresses, but the address they are trying their best to deliver spam to here never actually existed, of course. The first record of it at datadok.no was this bounce from a Russian site:

Jul 12 23:38:52 delilah spamd[29851]: (GREY) 81.177.34.190: 
<> -> <careersogt2083@datadok.no>

Dumping their trash back at them is good for a laugh, and I am quite amazed how shortsighted the spambot operators appear to be. They get yelled at for spamming, so to avoid detection, they start using fake addresses. This in turn means they have no feedback whatsoever on the quality of their address lists, and with well pissers like me in action, they are getting less effectitive each day, reducing themselves to background noise in the network.

Now with this blog post done I will go back and finish the edits on the logs chapter. With the early parts of the book about to enter the layout phase while the last bits get written over the next few days, there is a chance that there will be a physical copies of the book to pass around at OpenCON. Not quite there yet, but the fulltime push is certainly helping. The preface with a list of thanks is part of what is entering layout; I think a few people who did not expect to be in there will soon have a pleasant surprise.

Also this week, the PF tutorial saw its unique visitor number 19,000 since EuroBSDCon 2006 on Thursday morning (September 27th). We certainly hope at least some of them will come back for the book.

Wanna help science? Study your greylists' innards!

If somebody, say five years ago, had told me that I would be spending a little time, every day, studying data about what invalid addresses some unknown miscreants are making up in my domains, I would have thought them to be slighly off their rockers.

Yet here I am, actually maintaining a publicly available list of addresses which do not stand a chance of becoming valid, ever. It all started with a log data anomaly - I noticed an increase in the number of failed delivery messages to non-existent addresses in our domains. I had expected that the bounces to invalid addresses would appear for a short period only, but for one reason or the other it looks like it's here to stay, with some dips and peaks like the ehtrib.org flood.

The list is apparently working as intended too. These addresses are on my local greytrap list, and I have started seeing addresses I put in there as all uppercase turn up in my logs in all lowercase variants. Fun to watch, sort of.

Anyway, the supply of new bogus addresses proved to be larger than I had expected. So to get a handle on just what is happening I ended up doing periodic dumps of the live greylist data. This is really easy to do if you're using spamd as your greylister, your basic command is

$ sudo spamdb | grep GREY

and you redirect to a file, pipe to mail, or whatever you like.

Now if you're a bit like me, looking for patterns in the noise like this makes you feel a little weirder than usual and possibly lead you to think of a Clive Barker novel (specifically the bits about the dead letter file in The Great and Secret Show) and you wonder why this is worth doing at all. After all there is precious little spam that actuall reaches my users, so like I said earlier, for us spamd users it really looks like spam is a solved problem. I guess I'm just a bit fascinated by the pure irrationality of the spammers' behavior.

From the data I collect here in my tiny corner of the world to browse when time allows there may be useful information lurking somewhere.

Typical entries show things like the host 202.152.33.43 tried to send with a From: address jcejft@charter.com to dkqvujfn@datadok.no and sdenuuu@datadok.no. Using a few common networking commands we see that there is no reason why charter.com email should come from the IP range belonging to idola.net.id, and as the admin of datadok.no I know these two addresses have never been deliverable. Most likely the admin at charter.com can tell you if that from address is deliverable, but I keep wondering how much of the spam out there is stuffed into the pipe with bogus From: and To: addresses both. Or in other words, purely useless noise, never to be delivered anywhere.

On a side note, with one or more of the spammer operations trying to sneak through using sender and recipient addresses in the target domain, I assume it is just a matter of time before I see a tuple with both sender and recipient addresses already in my spamtraps list. When that happens, I think I will feel inclined to let my friends have a round of refreshments on my tab.

It's obvious that there are a handful of spammer operations that have decided to use datadok.no (and to a lesser extent, dataped.no and ehtrib.org) From: addresses on the spam they send, apparently in an attempt to cover their tracks. I will probably never know why they decided to do that, but I wonder why they keep it up and for that matter how many other domains are seeing this, with bounces from strange places, directed at non-existent, fairly obviously generated bogus addresses.

So if you are seeing similar stupidity in your logs and if you are running a sensible greylister such as spamd, I would be interested in hearing from you so we can compare notes.

Out there in meatspace, EuroBSDCon 2007 is coming up. I'll be there with the PF tutorial on Wednesday. This Friday's deadline for an updated manuscript had totally slipped my mind (I blame the book and a few other, less rational, factors), but hopefully the 24 who signed up for the session will find it useful anyhow - there will be new bits and as much interesting stuff as I can manage. I'll be around for the rest of the conference too, but unfortunately I'll have to give the Legonland trip a miss.

Be seeing you in Copenhagen! The book is getting closer to finished, I promise!

A Lady in Distress; or Then Again, Maybe Not

A two user domain gets bounces for seven hundred, grep and sed to the rescue, spamd saves the day

The past week moved along with only minor disturbances on the keep-systems-running front. The time consuming frustrations were generated elsewhere, and (un?)fortunately I am not at liberty to discuss the details. Incompetence was involved, next week it's somebody else's problem.

All the while, the spammer trapping experiment has been moving along at a leisurely pace.

Generally keeping the lists (both the web version and the live one) updated would cost me a few minutes' browsing of greylist dumps two or three times a day or whenever I felt like it, with a typical catch of maybe fifteen new bogus addresses to feed to the trap list each day.

For the last three or four days the haul has been smaller, with essentially no new captures yesterday, for example. Now I've found out why. They have moved on, alpabetically.

Done with bsdly.net, the dominant group of spammers moved on to generating addresses in the D domains including datadok.no and dataped.no. I'm bound to have missed a few, since the grand total by this morning had yet to reach a full thousand. By now, they seem to have reached the Es. This morning I noticed the overnight greylist dumps were bigger than usual.

The reason: ehtrib.org, the domain we set up mainly for my wife's use (read: her email), appears to be the current home of made up From: addresses, with roughly seven hundred accumulated by the time I was done with morning routines of breakfast with coffee and browsing the overnight incoming mail.

That is by far the largest addition to the flypaper list ever.

Fortunately, with only two active addresses in the domain (I'm not telling what either other one is) it's fairly trivial to extract the bogus ones.

Up to now I've been integrating the noise into the traplist page manually, for now I've put this batch up at http://www.bsdly.net/~peter/ehtrib-1stbatch.

They're all in the active traplist at the gateways, of course. It's the editing into the page the spammers will slurp via unattended robot I'm putting off for a little more while I'm doing some other writing. [not any more. all there now, but the original list is preserved too]

Just why this time we are seeing this number of addresses over a short period of time, and not a handful each day over several months is an open question. One likely explanation is that one of the chickenboners fell asleep at the wheel and let the junk generator run longer than they actually intended. Time will show if this means they move on more quickly.

When I have more time, I will probably analyse the data I am accumulating at the moment and tell the tales of the silly lamer tricks the spammers try to pull.

In the meantime, following up on earlier posts, there are still a few people who Just Don't Get It:

Aug 19 13:28:03 delilah spamd[3712]: 217.159.231.230: connected 
(9/9), lists: spamd-greytrap
Aug 19 13:31:49 delilah spamd[3712]: (BLACK) 217.159.231.230:
<> -> <armrest10@datadok.no>
Aug 19 13:33:32 delilah spamd[3712]: 217.159.231.230: Subject:
Considered UNSOLICITED BULK EMAIL, apparently from you
Aug 19 13:33:32 delilah spamd[3712]: 217.159.231.230: From:
"Content-filter at linux.byroomaailm.ee" 
<postmaster@linux.byroomaailm.ee>
Aug 19 13:33:32 delilah spamd[3712]: 217.159.231.230: To: 
<armrest10@datadok.no>
Aug 19 13:34:38 delilah spamd[3712]: 217.159.231.230: disconnected 
after 395 seconds. lists: spamd-greytrap

And it looks like the published list is having the effect I was hoping for. I keep seeing quite a few of the addresses in ALLCAPS (with numbers tacked on) I put on the web page a few weeks back beginning to appear in lowercase but otherwise identical in my greylist dumps.

In other news, the PF tutorial session at EuroBSDCon is now a definite.

See you in Copenhagen, if not before!

Now for that other bit of writing. The Book of PF page now refers to the tutorial page at bsdly.net. Now let's get that baby done.

The lady is, in fact, not too distressed.

And in case you were wondering - Yes, you can use my auto-generated list of trapped hosts for your own blacklisting purposes if you like. Here it's just a supplement to Bob Beck's traplist, and most likely you're better off using the Beck/UofA list along with your own greytrapping, but if you really want to use mine, be my guest. It gets updated ten past every hour.

On the business end of a blacklist. Oh the hilarity.

I had planned to write about something else for my next blog entry, but life came back and bit me with another spam related episode. Next time, I promise, I'll do something interesting.

In the meantime, I've discovered that a) very few people actually use SPF to reject mail b) the SPF syntax looks simple, but is hard to get right, and c) there are still blacklists which routinely block whole /24 nets.

This morning I got a message from somebody I met at BSDCan in May, asking me to do something LinkedIn-related. Naturally, since I felt I needed some more details to do what this person wanted, I sent a short email message. That message got rejected,

SMTP error from remote mail server after MAIL FROM: SIZE=2240:
    host mailstore1.secureserver.net [64.202.166.11]:
    554 refused mailfrom because of SPF policy

which means that the SPF record

 datadok.no. IN TXT "v=spf1 ip4:194.54.103.64/26 
                     ip4:194.54.107.16/29 -all"

does not do what you think it does. Mail sent from 194.54.103.66 was not let through.

OK, the checking tool at the OpenSPF site seem to agree with secureserver.net, and I seriously can not blame them for the choice to trust SPF absolutely.

At the moment it seems my listing each host name is what does the trick. Weird. Anyway, next up in my attempt to communicate with my overseas friend, I tried sending a message from bsdly.net instead. That bounced too, but for a slightly different reason:

 SMTP error from remote mail server after RCPT TO::
    host smtp.where.secureserver.net [64.202.166.12]:
    553 Dynamic pool 194.54.107.142. 

If you look up the data for bsdly.net, you will find that valid mail from there gets sent mainly from 194.54.107.19, which is in the tiny /29 our ISP set aside for my home net when I told them I wanted a fixed IP address.

I'm not sure if the rest of the "ip=194.54.107.*" network is actually a pool of dynamically allocated addresses these days, but I do know is that 194.54.107.16/29 has not been dynamically allocated for quite a number of years.

Going to the URL gave me this picture:



This really gives me no useful information at all. Except, of course, that at secureserver.net they think that putting entire /24 nets on their blacklist is useful. Some of us tend to disagree with that notion.

Anyway, I filled in the form with a terse but hopefully polite message, and clicked Submit.

I was rewarded with this message:



If I read this correctly, they think mail from 194.54.107.19 is spam because BGNett or MTULink have not set up reverse lookup for 194.54.107.142. OR because they think the entire /24 is dynamically allocated. OR somebody in that subnet may have sent spam at one time. I can only guess at the real reason, and repeat over and over that blocking entire subnets will give you a generous helping of false positives.

Nevermind that, the SPF record which made my mail from datadok.no go through to my overseas friend included a:hostname.domain.tld for all allowed senders.

And in other news, the PF tutorial saw its visitor number 15000 since EuroBSDCon 2006 on Saturday, last count is 15220.