![[en]](/_images/EN_flag_small.gif){kind=small}
![[us]](/_images/US_flag_small.gif){kind=small}
![[fr]](/_images/FR_flag_small.gif){kind=small}
#0x44: pSMTPD, goodbye hardcoded/implicit maps !
Tue, 07 09 2010 19:42, gilles@
pSMTPD has a map API which allows it to perform all kinds of key-based lookups. It was initially designed by pyr@openbsd.org, working in RAM, after what I extended it to also use flat files, db(3) and more recently an abstraction layer which allowed me to plug LDAP.
The maps are used everywhere, from the parsing of the configuration files to virtual users resolution or authentication to a remote MX. Whenever pSMTPD needs to perform a lookup, it does it through a map.
Initially, pSMTPD had a set of maps for which it expected specific names. For instance, aliases support was enabled only if there was a map named "aliases", the same applied for "virtual" and "secrets". This brought many limitations which is why I spent some effort to get rid of them. I ended up removing the last map expecting a specific name just three days ago when I turned the "secrets" map into a per-rule configurable parameter. A hard work but which allows for configurations that were not even possible before (read the previous blog entry for an example).
Yesterday evening, I decided to get rid of the last map that annoys me: the "localhost" map. It is hidden in parse.y and contains the list of IP addresses attached to all of the network interfaces the machine has. Whenever the rule set is read, rules containing no "from" or "from local" will consider this map to be the list of authorized sources.
It was a far more complex change than I assumed as it involved changing the ruleset matching logic a bit, but it is all for the better. I cannot commit it today as the change was invasive and I voluntarily removed big chunks of code in other places, but it will improve significantly the behavior of pSMTPD in some corner cases, while making quite a bit of code more readable.
A lot of cleanup is in progress, feel free to come by on #poolp @ irc.freenode.org if you want to follow up :-)
___
Tue, 07 09 2010 19:42, gilles@
pSMTPD has a map API which allows it to perform all kinds of key-based lookups. It was initially designed by pyr@openbsd.org, working in RAM, after what I extended it to also use flat files, db(3) and more recently an abstraction layer which allowed me to plug LDAP.
The maps are used everywhere, from the parsing of the configuration files to virtual users resolution or authentication to a remote MX. Whenever pSMTPD needs to perform a lookup, it does it through a map.
Initially, pSMTPD had a set of maps for which it expected specific names. For instance, aliases support was enabled only if there was a map named "aliases", the same applied for "virtual" and "secrets". This brought many limitations which is why I spent some effort to get rid of them. I ended up removing the last map expecting a specific name just three days ago when I turned the "secrets" map into a per-rule configurable parameter. A hard work but which allows for configurations that were not even possible before (read the previous blog entry for an example).
Yesterday evening, I decided to get rid of the last map that annoys me: the "localhost" map. It is hidden in parse.y and contains the list of IP addresses attached to all of the network interfaces the machine has. Whenever the rule set is read, rules containing no "from" or "from local" will consider this map to be the list of authorized sources.
It was a far more complex change than I assumed as it involved changing the ruleset matching logic a bit, but it is all for the better. I cannot commit it today as the change was invasive and I voluntarily removed big chunks of code in other places, but it will improve significantly the behavior of pSMTPD in some corner cases, while making quite a bit of code more readable.
A lot of cleanup is in progress, feel free to come by on #poolp @ irc.freenode.org if you want to follow up :-)
___
#0x43: pSMTPD supports per-rule credentials
Fri, 03 09 2010 15:39, gilles@
smtpd supports relaying via a remote MX rather than performing a DNS MX resolution. Some hosts require authentication before accepting to relay mail for the client.
Until now the credentials were stored in a hardcoded map called "secrets" and it was global to the entire setup. The credentials map would hold key values of the form:
relaymx.poolp.org user:password
And it would perform a first match so that if multiple relaymx.poolp.org keys were found, it would select only the first.
This is an issue as it prevents for instance the use of different credentials depending on the source machine or network. There was no way for example to allow mail from 192.168.1.0 and mail from 192.168.2.0 to be relayed via the same relay mx with different credentials.
I have just finished getting rid of the implicit "secrets" map and rules can now provide the map they want to look their credentials in, allowing the following:
map "servers-creds" { source db "/etc/mail/servers-creds.db" }
map "laptop-creds" { source db "/etc/mail/laptop-creds.db" }
accept from 192.168.1.0 for all relay via "mail.poolp.org" ssl auth "servers-creds"
accept from 192.168.2.0 for all relay via "mail.poolp.org" ssl auth "laptop-creds"
Since maps can use any backend, this paves the way for cool features such as having a db or ldap be dynamically updated.
___
Fri, 03 09 2010 15:39, gilles@
smtpd supports relaying via a remote MX rather than performing a DNS MX resolution. Some hosts require authentication before accepting to relay mail for the client.
Until now the credentials were stored in a hardcoded map called "secrets" and it was global to the entire setup. The credentials map would hold key values of the form:
relaymx.poolp.org user:password
And it would perform a first match so that if multiple relaymx.poolp.org keys were found, it would select only the first.
This is an issue as it prevents for instance the use of different credentials depending on the source machine or network. There was no way for example to allow mail from 192.168.1.0 and mail from 192.168.2.0 to be relayed via the same relay mx with different credentials.
I have just finished getting rid of the implicit "secrets" map and rules can now provide the map they want to look their credentials in, allowing the following:
map "servers-creds" { source db "/etc/mail/servers-creds.db" }
map "laptop-creds" { source db "/etc/mail/laptop-creds.db" }
accept from 192.168.1.0 for all relay via "mail.poolp.org" ssl auth "servers-creds"
accept from 192.168.2.0 for all relay via "mail.poolp.org" ssl auth "laptop-creds"
Since maps can use any backend, this paves the way for cool features such as having a db or ldap be dynamically updated.
___
#0x41: pSMTPD supports per-rule expiry
Fri, 03 09 2010 15:03, gilles@
Prior to the queue backout I had started working on per-rule expiry which would allow smtpd for example to retain messages with different expiry timeframes depending on their destination.
I just finished the feature to work on the old queue, allowing the following configurations:
accept for local deliver to mbox expire 16d
accept for all relay expire 32d
etc ...
While at it, I realized that the tagging of rules was broken as the configuration parser would simply ... not tag them. The fix was commited locally and sent to jacekm@ and nicm@ before it gets commited to OpenBSD.
___
Fri, 03 09 2010 15:03, gilles@
Prior to the queue backout I had started working on per-rule expiry which would allow smtpd for example to retain messages with different expiry timeframes depending on their destination.
I just finished the feature to work on the old queue, allowing the following configurations:
accept for local deliver to mbox expire 16d
accept for all relay expire 32d
etc ...
While at it, I realized that the tagging of rules was broken as the configuration parser would simply ... not tag them. The fix was commited locally and sent to jacekm@ and nicm@ before it gets commited to OpenBSD.
___
#0x40: poolp.org no longer runs OpenSMTPD
Fri, 03 09 2010 10:05, gilles@
Two months ago Jacek commited to OpenSMTPD a new queue code which would optimize OpenSMTPD's queue and make it better and faster amongst other bonuses.
Sadly I wasn't involved in the design of that queue and only got to discuss minor changes when most of it was already implemented. I okay-ed that diff because at that time it made things better and I didn't want to enter yet another debate.
It turns out that it also makes it almost impossible for me to implement various things I had planned and as much as I tried these last two months I am giving up.
I have just checked out a fresh copy of OpenSMTPD right before he commited his new queue code and merged all the local diffs I had (ldap support and various cleanups) to it. The result is a smtpd source that is different enough from OpenSMTPD that changes and features I will implement will be hard, if not impossible, to port.
poolp.org no longer runs OpenSMTPD, it runs its own version which I will rename to psmtpd for short.
___
Fri, 03 09 2010 10:05, gilles@
Two months ago Jacek commited to OpenSMTPD a new queue code which would optimize OpenSMTPD's queue and make it better and faster amongst other bonuses.
Sadly I wasn't involved in the design of that queue and only got to discuss minor changes when most of it was already implemented. I okay-ed that diff because at that time it made things better and I didn't want to enter yet another debate.
It turns out that it also makes it almost impossible for me to implement various things I had planned and as much as I tried these last two months I am giving up.
I have just checked out a fresh copy of OpenSMTPD right before he commited his new queue code and merged all the local diffs I had (ldap support and various cleanups) to it. The result is a smtpd source that is different enough from OpenSMTPD that changes and features I will implement will be hard, if not impossible, to port.
poolp.org no longer runs OpenSMTPD, it runs its own version which I will rename to psmtpd for short.
___
#0x3f: What's new pussy cat ? whoooo ooooh whoooo oh ooooooh !
Mon, 30 08 2010 14:22, gilles@
Meow !
After three weeks of vacations far away from computers-land, I got back to working a bit on my projects.
Spent end of last week writing a prototype project for a customer. PHP/Ajax/MySQL, stuff I usually hate but which turned out to be fun this time :-)
I realized that my last commit to PCT was actually missing a file preventing compilation to succeed. I will pick up on it this week, when I took off for vacation I had initial support for RSA signing/verification, so it shouldn't take long before I can sign/verify using keys from keyrings.
I am currently working on poolp's smtpd, my plan is to introduce new features and get rid of some code I dislike. Sadly, the new queue code brought by jacekm@openbsd.org makes it hard for me to implement any feature so it is very likely I'll revert to the old queue code in poolp's repository. This will mean that smptd will diverge a lot between OpenBSD's tree and mine making it almost impossible to backport diffs from one to another ... still trying to avoid that but I'm about to give up.
I am slowly getting familiar with Android hacking, been playing a bit with creating my own roms out of stock boot.img, took a few hours to explore the various system files and will give a try at building my own kernel/rom this week. Just for the record, I managed to make a rom with stock boot.img, with my own custom /system shipping python 2.6 and some custom scripts. It is fun :-)
Voila, stay tuned for poolp news !
___
Mon, 30 08 2010 14:22, gilles@
Meow !
After three weeks of vacations far away from computers-land, I got back to working a bit on my projects.
Spent end of last week writing a prototype project for a customer. PHP/Ajax/MySQL, stuff I usually hate but which turned out to be fun this time :-)
I realized that my last commit to PCT was actually missing a file preventing compilation to succeed. I will pick up on it this week, when I took off for vacation I had initial support for RSA signing/verification, so it shouldn't take long before I can sign/verify using keys from keyrings.
I am currently working on poolp's smtpd, my plan is to introduce new features and get rid of some code I dislike. Sadly, the new queue code brought by jacekm@openbsd.org makes it hard for me to implement any feature so it is very likely I'll revert to the old queue code in poolp's repository. This will mean that smptd will diverge a lot between OpenBSD's tree and mine making it almost impossible to backport diffs from one to another ... still trying to avoid that but I'm about to give up.
I am slowly getting familiar with Android hacking, been playing a bit with creating my own roms out of stock boot.img, took a few hours to explore the various system files and will give a try at building my own kernel/rom this week. Just for the record, I managed to make a rom with stock boot.img, with my own custom /system shipping python 2.6 and some custom scripts. It is fun :-)
Voila, stay tuned for poolp news !
___
#0x3e: Quick news...
Wed, 18 08 2010 19:55, gilles@
Just a quick post to thank Philippe Chataignon for the donation of a Sunblade 1500 which will help me work on my projects.
I like the sparc64 architecture because it is different enough from i386/amd64 and a bit more strict in some aspects. It helped me many times spot alignment issues which had been sitting for weeks unnoticed on my amd64.
Until now I worked on a sunblade 100 which was ok and very helpful but a bit too slow for my needs. The sunblade 1500 brings me to the world of Ghz CPU allowing me to perform CPU intensive tasks twice as fast as before (no more time to go make a coffee when I generate a RSA key for instance ;)
Many thanks again !
My blog will be a bit calm the next two weeks, I am currently on vacation, been away ten days and very likely going away on sporadic journeys in my new city and its neighbours ;)
___
Wed, 18 08 2010 19:55, gilles@
Just a quick post to thank Philippe Chataignon for the donation of a Sunblade 1500 which will help me work on my projects.
I like the sparc64 architecture because it is different enough from i386/amd64 and a bit more strict in some aspects. It helped me many times spot alignment issues which had been sitting for weeks unnoticed on my amd64.
Until now I worked on a sunblade 100 which was ok and very helpful but a bit too slow for my needs. The sunblade 1500 brings me to the world of Ghz CPU allowing me to perform CPU intensive tasks twice as fast as before (no more time to go make a coffee when I generate a RSA key for instance ;)
Many thanks again !
My blog will be a bit calm the next two weeks, I am currently on vacation, been away ten days and very likely going away on sporadic journeys in my new city and its neighbours ;)
___
#0x3d: Gilles: Summer 2006
Sat, 31 07 2010 12:00, gilles@
Second solo tune, "Summer 2006" was inspired by the massacre of Qana and the slaughter that occured in Beiruth and South Lebanon during the Summer 2006.
In Qana, a building where children and women hid from the combats was bombed by the israeli forces. During the summer, over 1000 civilians were massacred, Europe did nothing to stop that and USA encouraged israel to continue.
The tune contains real recordings of bombings from back then, as well as extracts from news reports and interviews.
Feel free to download, share and let me know if you like it. I have a webpage at My Major Company, should you want to sponsor me ;-)
___
Sat, 31 07 2010 12:00, gilles@
Second solo tune, "Summer 2006" was inspired by the massacre of Qana and the slaughter that occured in Beiruth and South Lebanon during the Summer 2006.
In Qana, a building where children and women hid from the combats was bombed by the israeli forces. During the summer, over 1000 civilians were massacred, Europe did nothing to stop that and USA encouraged israel to continue.
The tune contains real recordings of bombings from back then, as well as extracts from news reports and interviews.
Feel free to download, share and let me know if you like it. I have a webpage at My Major Company, should you want to sponsor me ;-)
___
#0x3a: Gilles: Ghaza
Tue, 27 07 2010 22:06, gilles@
Be it for OpenBSD, smtpd, poolp or random hacks, most of my post are related to computer science somehow.
I also happen to be a musician and after moving out from Paris and leaving my band there, I decided to try and join or start a few projects here in Nantes. One of these project is a solo project which I am currently working on...
First tune to be released, Ghaza, was inspired by the death of 9 activists on the Marvi Marmara, in 2010.
Feel free to download, share and let me know if you like it. I have a webpage at My Major Company, should you want to sponsor me ;-)
___
Tue, 27 07 2010 22:06, gilles@
Be it for OpenBSD, smtpd, poolp or random hacks, most of my post are related to computer science somehow.
I also happen to be a musician and after moving out from Paris and leaving my band there, I decided to try and join or start a few projects here in Nantes. One of these project is a solo project which I am currently working on...
First tune to be released, Ghaza, was inspired by the death of 9 activists on the Marvi Marmara, in 2010.
Feel free to download, share and let me know if you like it. I have a webpage at My Major Company, should you want to sponsor me ;-)
___
#0x39: PCT: Poolp Crypto Toolkit
Mon, 26 07 2010 21:02, gilles@
I have renamed the nbpg project into PCT, which stands for Poolp Crypto Toolkit.
Not so many visible improvements, I did quite a bit of cleanup and wrote a small allocator which relies on mmap() and will help me spot bugs and general memory usage. It also allows for a complete wipe of allocated chunks preventing informations leak in case of a memory leak or in case a *common* error path is followed.
[...]
p1 = pctalloc(1, sizeof(*p));
p2 = pctalloc(1, sizeof(*p));
p3 = pctalloc(1, sizeof(*p));
p4 = pctalloc(1, sizeof(*p));
pctfree(p1); // zeroes and releases p1
pctwipe(); // zeroes and releases ALL allocated chunks
[...]
___
Mon, 26 07 2010 21:02, gilles@
I have renamed the nbpg project into PCT, which stands for Poolp Crypto Toolkit.
Not so many visible improvements, I did quite a bit of cleanup and wrote a small allocator which relies on mmap() and will help me spot bugs and general memory usage. It also allows for a complete wipe of allocated chunks preventing informations leak in case of a memory leak or in case a *common* error path is followed.
[...]
p1 = pctalloc(1, sizeof(*p));
p2 = pctalloc(1, sizeof(*p));
p3 = pctalloc(1, sizeof(*p));
p4 = pctalloc(1, sizeof(*p));
pctfree(p1); // zeroes and releases p1
pctwipe(); // zeroes and releases ALL allocated chunks
[...]
___
#0x38: nbpg moving forward
Fri, 23 07 2010 20:37, gilles@
I had temporarily paused nbpg development because I was facing a bug that I did not have much time to investigate.
Whenever parsing a Compressed Data Packet, nbpg would successfully extract the data but fail to inflate it with a data format error. No matter what I tried, I could not figure out what was wrong with my code or with the data.
Turns out that I had to rely on an undocumented feature of zlib. I discovered it when I gave up at trying to make RFC interpretations and looking at what the GnuPG guys did. I saw the following comment in their compress.c:
* PGP uses a windowsize of 13 bits.
* it forces zlib not to expect a zlib header. This is a
* undocumented feature Peter Gutmann told me about.
After doing a one line diff to nbpg I managed to extract a compress data packet. This allowed me to parse two new packets and I can now move things forward :-)
Output of a signed text file:
### 0x20cd54000: Compressed Data packet
length: 107
algorithm: ZIP
### 0x20cd54800: One-Pass Signature packet
length: 13
version: 3
type: Signature of a binary document
hash algorithm: SHA-1
public-key algorithm: DSA
keyid: F838AC47FA273AD7
nested: no
### 0x20cd55000: Literal Data packet
length: 19
type: binary data
filename: bleh.txt
flags: none
buffer: [test
]
### 0x20cd55c00: Signature packet
length: 70
version: 4
type: Signature of a binary document
public key algorithm: DSA
hash algorithm: SHA-1
size of hashed area: 6
size of unhashed area: 10
___
Fri, 23 07 2010 20:37, gilles@
I had temporarily paused nbpg development because I was facing a bug that I did not have much time to investigate.
Whenever parsing a Compressed Data Packet, nbpg would successfully extract the data but fail to inflate it with a data format error. No matter what I tried, I could not figure out what was wrong with my code or with the data.
Turns out that I had to rely on an undocumented feature of zlib. I discovered it when I gave up at trying to make RFC interpretations and looking at what the GnuPG guys did. I saw the following comment in their compress.c:
* PGP uses a windowsize of 13 bits.
* it forces zlib not to expect a zlib header. This is a
* undocumented feature Peter Gutmann told me about.
After doing a one line diff to nbpg I managed to extract a compress data packet. This allowed me to parse two new packets and I can now move things forward :-)
Output of a signed text file:
### 0x20cd54000: Compressed Data packet
length: 107
algorithm: ZIP
### 0x20cd54800: One-Pass Signature packet
length: 13
version: 3
type: Signature of a binary document
hash algorithm: SHA-1
public-key algorithm: DSA
keyid: F838AC47FA273AD7
nested: no
### 0x20cd55000: Literal Data packet
length: 19
type: binary data
filename: bleh.txt
flags: none
buffer: [test
]
### 0x20cd55c00: Signature packet
length: 70
version: 4
type: Signature of a binary document
public key algorithm: DSA
hash algorithm: SHA-1
size of hashed area: 6
size of unhashed area: 10
___
![[openbsd]](/_images/pb-openbsd.png){kind=small}
![[apache]](/_images/pb-apache.png){kind=small}
![[python]](/_images/pb-python.png){kind=small}