Poolp

DSN is a standart way of representing status codes so that they are easily understandable by a human AND a machine. For instance, many errors will trigger a status code of 530 which will let the client know there's a permanent error, but which will not let it interpret what caused the error. DSN adds an additionnal status which is represented by three digits, describing a class (success, temporary error, permanent error), a subclass (mailbox, address, etc) and an additionnal digit to distinguish between the various messages that could fit a class and a sublass of success/errors.

For example, instead of having: "250 Sender ok", we'd have "250 2.1.0 Sender ok" with 2 being a success, 1 an address related message and 0 being the sender ok specific message.

[...]
MAIL FROM:
250 2.1.0 Sender ok
[...]

This is a work in progress, it just allows some mua's to do a better job. It is not ready to be commited.

OpenSMTPD has no restriction on the size of a message and will accept DATA content to be as long as can fit in the disk slice holding the queue.

I just implemented support for the RFC1870 SIZE extension which let's the server advertise the maximum size allowed for a message at EHLO time, then reject with a permanent error any message exceeding it or overflowing a size_t.

Administrator can now set a maximum message size in smtpd.conf using the following syntax:

size 1024 # 1024 Bytes
size 1 KB # 1024 Bytes = 1 KB
size 1 MB # 1024 KB = 1 MB
size 1 GB # 1024 MB = 1 GB

A session will advertise:

$ nc localhost 25
220 vaio.poolp.org ESMTP OpenSMTPD
EHLO L
250-vaio.poolp.org Hello L [127.0.0.1], pleased to meet you
250-8BITMIME
250-SIZE 2147483648
250 HELP
$

And will fail with:

554 Transaction failed

If message exceeds 2GB in this example.

Not providing the "size" keyword in smtpd.conf will default to MAX_SIZE, the maximum value for a size_t, and will be way enough for you to continue exchanging movies through OpenSMTPD

Oh, it's not commited yet ...

OpenSMTPD supposedly had netmask matching capabilities in rulesets, however I mainly used implicit /32 and implicit /0 masks in my own setups and, apart from a few tests using /8, /16 and /24 masks which happened to work by chance, I never did much testing on netmasks.

nicm@ reported a misbehaviour a while ago and enclosed a diff which turned out to cause a regression, after a bit of investigation and updates I cooked another diff which fixed his regression but introduced another, then the bug hit my TODO and was forgotten till today. nicm@ asked me if I had a chance to look at the bug, reminding me that I should spend some time looking for it. We both did a bit of testing and thinking and after trying to fix the matching function, it turned out parse.y was partly busted using a different meaning for the same member of a struct depending on the context. This would cause the matching function to work as expected for some setups and not for others, and the more we tried to fix these setups, the more we broke the others.

I came up with a fix with lots of help from nicm, unbreaking AF_INET netmasks effectively. I'll do some more testing and commit it, I just can't wait to do the same work on AF_INET6 ...

Juste un petit post pour prévenir qu'une install party aura lieu dans les locaux d'Epitech Nantes le Samedi 13 Mars 2010, à partir de 10h.

Ceux qui veulent participer pourront venir avec leur machine pour se faire installer ou aider les autres à s'installer un système libre parmi plusieurs systèmes BSD ou distribs Linux.

Les locaux se situent au 18 rue Flandres Dunkerque 1940, 44100 Nantes, près du Muséum d'Histoire Naturelle et du Musée Dobrée.

L'évènement est ouvert aux externes

Gilles

As I mentionned in my last post, part of the backend-agnostic maps was commited to OpenBSD's tree but they are now fully working and I moved things forward a bit to improve reliability.

Prior to that commit, maps were not really handled in a sane way. We'd store structs in aliases db and plaintext in secrets db, makemap being in charge of deciding what gets written and how based on a getopt flag. Each time we made a change to struct alias or to any of the struct it had as a member, the aliases db had to be rebuilt. On smtpd's side, the fact that we stored struct was not really helping as we still had to perform sanity checks which eventually cost as much as parsing the original alias entry into a struct alias.

Jacek and I discussed a while ago about a feature we wanted to have, where the map API would be aware of some data types we used through smtpd. For instance, since the aliases feature uses a struct alias, we should be able to have the map API return it instead of returning a string that the aliases API would parse. The code could then work with known structures and it would avoid bugs spreading in various places.

I came up with an idea that each "feature" would provide a simple parse function to turn a plaintext value into a struct for that feature. For example, autenticated relaying support provide a map_parse_credentials() function which parses a plaintext input of the form "user:password" into a struct map_credentials containing two members, one storing the username, the other storing the password. Since all backends store a plaintext representation, the parser will work no matter what backend is used. If a particular feature needs the raw value, it simply needs to mention it does not have a parser and things will work out of the box.

Prior to this, to retrieve credentials, the lka process had to call map_lookup(), retrieve the value, check if it was not NULL, check if value was correctly formed, fill a struct with it. Aliases was even worse since a value can in fact contain many values and each one can be parsed differently. Now, one can simply:


struct map_credentials *creds = map_lookup(env, map, "gmail.com", K_CREDENTIALS);
if (creds == NULL) {
// no credentials
}
else {
// creds->username and creds->password are guaranteed to be correct
free(creds);
}


same applies for aliases/virtual (or soon at least since i'm mostly done) and any feature requiring maps lookup.

Hopefully this can go in soon as it will help me cleanup the aliases resolution code path for good

The map API is in charge of providing OpenSMTPD with a set of functions to easily lookup key/value maps or one-column sets. It is used in various areas, ranging from aliases resolution to virtual domains or even remote authentication credentials (to have OpenSMTPD relay via your provider's MX hosts after authenticating, for example). maps can be ram-based (runtime maps) or disk-based (persistent maps), which implies the use of a backend to store the information between restarts.

Until now, we assumed the backend to be db(3) databases and this was kind of annoying because some people want to use flat files for fast changes, while others need to have the lookups done over a sql or ldap database. Not to mention that a SQL backend provides many features outside the scope of OpenSMTPD that can make its use way sexier...

I rewrote the map API to be backend-agnostic, which basically means that there is no longer any assumption about the backend used. Now, OpenSMTPD comes with two "official" backends, db(3) databases and stdio(3) which allows the use of flat files, but the maps API has a mechanism which will allow us to very easily add support for new backends without too much pain. To add support for a new backend, one has to provide a set of 4 callbacks which will open the backend, lookup a key from the backend, write a key/value to the backend, close the backend. Both db(3) and stdio(3) were implemented in very few lines and I expect pgsql/mysql/ldap not to be much more difficult.

Today I commited the changes that make the map API backend-agnostic but without enabling support for stdio(3), the other changes will be commited after the src tree is locked and unlocked because they will force a rebuild of aliases databases, will probably force a queue flush and introduce at least a few regressions I did not even think of yet

After taking a rather long break from OpenSMTPD, which stayed a bit active thanks to jacekm@, I started getting back to work and giving some ideas a deeper thought. I'm currently working on the map API which I started redesigning a bit after discussing it with Jacek a couple weeks ago. I will not commit bits before the src lock that's going to happen in not too long, but there will be active development in that area during March.

Will keep you informed very soon

As some of you know, I have recently left my job and life in Paris to start a new life in Nantes, not so far away from the Atlantic coast.

I am currently employed at Epitech, european institute of technology, a private computer science school where I assist the director of studies teaching various classes and dealing with administrative tasks. It's fun, I get to meet lots of interesting people and help students get familiar with C, Unix and related, I like it a lot.

The work intensity varies depending on periods which means that when students are on vacation or working hard on their projects before a due date, I can leave early and get enough free time to hack on my own stuff. Last week, for instance, I managed to write a device driver for an USB/IrDA adapter, which I have been willing to do for months without ever finding the time to do it:

uircom0 at uhub5 port 2 configuration 1 interface 0 "DeLock USB IrDA adapter" rev 1.10/0.08 addr 2
ucom0 at uircom0, Generic

With regard to Poolp, I am investigating and experimenting on services and infrastructure changes but there's nothing new that I can foresee in a close future. After all, it's been working great for years now with only minor glitches when we switched from sendmail to opensmtpd. I am currently working on projects with partners, stuff I can't talk about yet but which could end up injecting money into poolp to expand it.

OpenSMTPD development has slowed down a bit but it is still active and both jacekm@ and I are working on parts of it. The fact is we both have diffs which change big chunks of it and cannot commit them yet as the OpenBSD tree will be locked in a few weeks, not long enough for us to make sure we don't cause a major regression.

Anyway, I'll try to keep the blog updated a bit more often now that I have a couch I can sit on in my huge castle

Not so many news coming from me these last days, but I've been working quite a lot

On the OpenSMTPD side, after porting it to FreeBSD and NetBSD with the help of various people, I started porting to Linux to at least reduce the delta.
I managed to get a good share of OpenSMTPD building on an Ubuntu system with minor changes here and there which is pretty good.
The changes aren't commited yet, they should be in coming weeks.

On the poolp side, mostly administrative tasks and forms to fill.
Experiments have been running to provide members with VPN access as well as Python&Django for dynamic content.

More news coming soon

In a few days will be commited a huge diff to the OpenBSD tree which changes some of the data structures and improves many areas. It has kept me busy for days and it still needs some work, I'll explain the changes in a later post to this blog, when the diff actually gets in.

Meanwhile, I decided to simplify the porting of OpenSMTPD to other systems by having both BSD make and the `mk' framework dependencies to build OpenSMTPD on other systems. After all, I've been forced to install gmake on every single BSD system I ever administered, because it is a dependency to build almost all applications coming from the GNU world, I guess having BSD make a dependency for building a BSD application in the GNU world isn't such a heavy requirement

Expansion is the operation by which OpenSMTPD resolves a session recipient into delivery recipient(s).
Currently there are various expansions taking place:

- ~/.forward files expansion
- aliases expansion
- virtual expansions

And it all becomes more and more tricky when we get to cross-references where an alias points to multiple local users, some being aliases, some not, and some having ~/.forward files eventually pointing to other aliases. As you can imagine, it is not a trivial operation and there are many issues that arise.

Until recently, OpenSMTPD was using a naive approach which consisted in using a structure for each single expansion node that was being loaded from disk to memory and pushed inside a queue. The expansion function would perform 5 iterations over the queue, adding new results at the end of this queue. This was actuallly done to provide basic support for aliases until we came up with a proper implementation, and it had many drawbacks: duplicate nodes would be stored many times in memory and part of the unique queue which we'd scan iteratively 5 times. I think I saw a big O notation cry ...

Yesterday I started a serie of rewrites to replace our current implementations with a new, smarter, one. The expansion now uses two structures, one disk-based, used to represent an alias for example, and one ram-based that can be built from the disk-based structure, and which lets us store useful informations during the expansion. We know have a deliverylist, which is a queue of recipients (both local and remote) we consider final and which no longer need any kind of work, and we have an expandtree which is a red-black tree of expand_nodes, which contain the data to expand other expand_nodes or final recipients.

The new design is still an iterative process, but instead of storing one node for each entry we want to expand, it stores one node with a reference count and a flag. expanding a node flags it as F_EXPAND_DONE and loops will increase the reference count. This has many benefits: loops no longer grow memory usage, duplicates are handled automatically both in terms of memory usage and reference usage, but also in terms of processing. The same node will expand identically, so if we find a node, insert it in the tree and see that it is flagged F_EXPAND_DONE, it means that we already did the processing and can skip it. Finally, it allows us to detect that we're done with the expansion earlier. If all of our nodes are F_EXPAND_DONE, then we know that another iteration is not going to bring new results and we can proceed to deliveries.

This all came with various cleanups and is still a work in progress, it's been commited to OpenBSD's tree as of today and it probably broke portable OpenSMTPD's changeset which I'll sync tomorrow most likely.

OpenSMTPD reintroduces support for the '+' escape character after it's been removed during the virtual domains expansion rewrite.

The feature allows a local user to append information to the user-part of his email address that will not be processed by smtpd when it receives mail for that user. To provide a clearer example, imagine that whenever I send a mail to misc@openbsd.org, I submit the following addresses: gilles+misc@poolp.org. Whenever someone writes back, OpenSMTPD parses the recipient address as:

user: gilles+misc
domain: poolp.org

Until now, this would fail because gilles+misc is not a valid user, but with the new code OpenSMTPD extracts the username out of the userpart by taking the part that preceeds '+' so that:

user: gilles+misc
domain: poolp.org
pw_name: gilles

The user part remains untouched, gilles+misc is considered to be the local recipient but when we attempt to open a mailbox or access the ~/.forward file, we use the pw_name which holds the real system user.

Fixing it was actually about 5 lines of code, it is already in use at poolp.org where users can use random suffixes, and will be commited in the OpenBSD tree later tonight.

PHP has been updated to version 5.2.11 on the primary server.
PostgreSQL has been installed on the primary server and should be installed as a replication slave on the secondary server soon.

Jacek Masiulaniec is featured in a podcast interview where he talks about the OpenSMTPD project:

http://bsdtalk.blogspot.com/2009/11/bsdtalk179-openbsd-developer-jacek.html

Enjoy !

As you probably figured by now, poolp.org uses OpenSMTPD to provide mail services to its users. A few months ago, I received a mail which was kind of broken: my mail client would not display the sender and subject, and the body of the mail contained many of the headers, including From: and Subject:. I realized that an empty line had been inserted in the middle of headers, causing the mail user agent to choke on the message.

Given that OpenSMTPD writes directly to the mailbox the data that client sends without any kind of rewrite, I was confident this was no bug from us. The fact that only one mail I received out of thousands in months had this issue was enough to convince me this was a client problem. Weeks passed before Matthieu Goessens, a poolp.org user, complained that he had received a broken mail which displayed part of the headers in the body. After that mail, he confirmed receiving them more frequently, one every few days, from the same mailing list and with the same users being able to trigger the bug every time they sent mail.

I started adding debug logs everywhere and it became obvious that there was something strange and funky happening inside OpenSMTPD. The mails would arrive on my network without that empty line, but very rarely as they were read using the evbuffer_readline() function from libevent, an empty line would be returned after a line was read correctly. And to make things worse, it was impossible for me to reproduce the bug, even by replaying the session myself.

After reading the evbuffer_readline() logic, it appeared to be correct and unlikely to be bugged as it was used in many other applications. I ended up adding debug output all over the place, printing the offset of every single \r and \n, printing the buffers and rewriting my own evbuffer_readline() replacement with a slightly simpler code and logic.

Today I received another broken mail and finally understood what had happened...

evbuffer_readline() from libevent assumes \r, \n, \r\n, \n\r and \r\r to be end of line delimiters. the SMTP protocol uses \r\n as the end of line delimited.
What happened is the following: the client sent a long line and the server did not read it at once, but read a chunk, then another.

client sends: "verylooooooooooooooo\r\nooooooooooooooooooooongline\r\nanoooooooootherlongline\r\n"
server reads: "verylooooooooooooooo\r\nooooooooooooooooooooongline\r" "\nanoooooooootherlongline\r\n"

libevent fills a buffer with the first chunk, notifies OpenSMTPD that there's data in the buffer. OpenSMTPD calls evbuffer_readline() in a loop causing us to read:

buffer: "verylooooooooooooooo\r\nooooooooooooooooooooongline\r"
first evbuffer_readline() call: "verylooooooooooooooo"
second evbuffer_readline() call: "ooooooooooooooooooooongline"

then returning. This is where the error lies. since SMTP protocol expects \r\n, the last line should not have been read because the buffer only ends with \r. evbuffer_readline() assumes this is an end of line delimiter but it is not true for OpenSMTPD. When we return, libevent gets a chance to read the other chunk and add it to the buffer which means that now evbuffer_readline() sees:

buffer: "\nanoooooooootherlongline\r\n"
first evbuffer_readline() call: ""
second evbuffer_readline() call: "anoooooooootherlongline"

The first call hits an end of line delimiter immediately and returns an empty line when it was just a part of the the end of line delimiter for the buffer we read earlier. I had suspected a bug of this kind so I had written a evbuffer_readline() replacement which was more strict on its check, but I did not want to commit it until it was proven that this was really the cause of the bug. And given how rare the broken mails were flowing, I had to stick with the "broken" version and some debug code to prove that my version produced a different and valid result compared to the original evbuffer_readline().

The problem here was actually a combination of problems really. It is because evbuffer_readline() did not have the exact semantic we needed AND the data was written from the client to the server in such a way that it got splitted at the very wrong place, that the bug triggered. I'd say we're even pretty lucky that it has triggered so often given how low the odds were to hit that case.

OpenSMTPD now uses its own evbuffer_readline() variant with strict semantic to avoid issues of this kind. Bug fix has been commited today into the OpenBSD tree and service on poolp.org has been updated.