ANNOUNCE: Haraka 2.8.14

(I’m a bit late on this one – it has been out for a few days)

This release has built into a very large release due to much going on in the lives of those involved. It includes some major changes (hopefully none that break too much) so please release to your production servers with the usual caution.

2.8.14 – Jul 26, 2017

• Changes
  • Fix auth plugin failure when re-selecting auth method #2000
  • don’t crash Haraka when invalid YAML config encountered #2013
  • update semver to version 5.4.0 #2015
  • relay docs: correct the config file name #2012
  • rename config/xclient.hosts to match plugin & docs #2014
  • build_todo() is part of the outbound/index.js api #2016
  • update js-yaml to version 3.9.0 #2002
  • outbound/hmail: use WRITE_EXCL from haraka-constants #2011
  • replace plugins/log.elasticsearch with npm packaged #2004
  • Remove two spurious log statements #1989
  • access: rebuild blacklist upon change (vs supplement) #1990
  • deliver to qmail-queue with LF line endings (not CRLF) #1997
  • doc: add note that smtp_forward only supports STARTTLS #1988
  • import Plugins.md from v3 #1991
  • update async to 2.5.0 #1982
  • update iconv to 2.3.0 #1981
  • require node.js v6+ #1958
  • update ipaddr.js to 1.4.0 #1972
  • support newer address-rfc2822 #1970
  • update node-address-rfc2821 version to 1.1.1 #1968
  • outbound: be consistent with todo.domain #1960
  • bump haraka-results required version #1949
  • logger: load in a setImmediate call #1948
  • logger: strip intermediate \n chars #1947
  • tls consistency cleanups #1851
  • Get pool config handling simplifcation #1868
    • add integration test: send message w/smtp_client
  • replace some legacy code with es6 #1862
  • update async to version 2.2.0 #1863
  • update ipaddr.js to version 1.3.0 #1857
  • update redis to version 2.7.0 #1854
  • assure conn/tran still exists before storing results #1849
  • moved tls.ini parsing to net_utils #1848
  • smtp forward dest split routing #1847
  • rspamd: refactor complex condition into function #1840
  • block js attachments #1837
  • helo.checks: bring plugin into alignment with docs #1833
  • when proxy enabled, update remote.is_private too #1811
  • create an outbound queue filename handler #1792
  • replace connect.fcrdns with npm package #1810
  • add an additional node_modules plugin search path #1805
  • Set graceful shutdown off by default #1927
  • Allow outbound pools to be disabled #1917
  • Outbound split and move into folder #1850
  • don’t emit binary characters into the logs #1902
  • Add .editorconfig #1884
  • tls: remove interim variables #1871
• New Features
  • Use punycode domain (support SMTPUTF8) #1944
  • Added RabbitMQ vhost support #1866
  • clamav: allow “Unknown Result” and Socket Error to try next host #1931
  • outbound client certificates #1908
  • Implement the missing upgrade method on SMTPClient #1901
  • Remove typo from relay.md #1886
• Fixes
  • outbound: fix queue not loaded for single process #1941
  • outbound: Fix undefined variable platformDOT in hmail.js #1943
  • outbound: fix undefined FsyncWriteStream var #1953
  • Fix cluster messaging for node v6+ #1938
  • outbound: fix loading under cluster. #1934
  • Check pool exists before delete #1937
  • be more strict in attachment filename matching #1957
  • doc typo fix #1963
  • RabbitMQ: fix encoding of user and password string #1964
  • spf: improve modifier regexp #1859
  • rabbitmq doc typo in config file name #1865
  • URL to manual was 404, point to Plugins.md #1844
  • smtp_client: set idleTimeout to 1s < pool_timeout #1842
  • fix broken continuations #1843
  • doc error for the ‘check.authenticated’ setting in rspamd plugin #1834
  • emit the result, not all of them #1829
  • fix outbound logger #1827
  • fix forwarding with client auth over TLS (forward to gmail) #1803
  • Don’t blow the stack on qstat #1930
  • run dumped logs through log plugins, not console #1929
  • Fix path parsing bug on Windows platform #1919
  • helo: make sure list_re is defined before access #1903
  • TLS: handle case where OCSP server is unavailable #1880
  • rspamd: add missing ‘default’ keyword #1856
  • disable naïve comment stripping #1876

ANNOUNCE: Haraka v2.8.11

I’m a fair bit late on this one – 2.8.11 has been on npm for a while. My apologies to the team for letting a blog post and email slip. There’s been some more fantastic changes in this release related to TLS, specifically adding OCSP stapling. Full change list below.

• Changes
  • rename core_require to haraka_require #1708
  • move log.syslog to haraka-plugin-syslog #1698
  • remove tls.ini loading and is_no_tls_host to net_utils #1690
  • replace ./utils with npm packaged haraka-utils #1720
  • require node 4
  • karma: add .top TLD scoring #1714
• New Features
  • Implement OCSP Stapling #1724
• Improvements
  • show help for npm packaged plugins included in core #1698
  • use tls.connect for client #1682
  • bring port 465 SMTPS TLS config support on par with STARTTLS #1667
  • use tls.connect instead of createSecurePair #1678
  • redis: improve error handling in tests #
  • replace / path seperators with path.* for cross platform compat #1713
• Bug Fixes
  • dkim_sign: per-domain key finding fixed #1707
  • Rspamd: restore spam report header #1702
  • auth/vpopmail: do not toString() when null #1695
  • fix outbound to avoid recursive reading key/cert after refactoring #1692
  • tls: fix option servername (not hostname) #1728
  • correct Auth-Results cleaning #1726
  • fix results for connection.remote_host and NXDOMAIN #1716

ANNOUNCE: Haraka 2.8.10

This release pretty much signifies exactly why I love truly open source software – the release happened almost entirely in my absence, and I’m enormously grateful for the talented developers that work so hard and diligently on this project.

The main focus of this release has been some huge improvements to our TLS support from a contributor only known as “typingArtist” – ah the mysteries of the internet – we may never know your real life identity but we remain grateful – you are Haraka’s “Batman” this release. Many further thanks to Matt Simerson who coordinated this release, and continues to contribute fantastic work to this project. We don’t forget our other contributors, but those two were the stand outs in this release.

• Changes
  • use standard npm syntax for lint and tests #1646
  • remove ./net_utils to haraka-net-utils #1644
  • remove incorrect and unused spf.hello_host #1635
  • remove rogue DENYSOFT copy-pasta error #1634
  • update async to v2 #1545
  • remove plugin/dir support from base haraka #1668
    • use node_modules_dir support instead
  • use TLSSocket instead of createSecurePair #1672
  • refactor plugins/tls #1670
  • moved watch plugin to npm as haraka-plugin-watch #1657
  • normalize proxy properties #1650
• New Features
  • added connection.remote.is_private boolean #1648
  • added additional TLS options (@typingArtist) #1651
  • added wildcard boolean support to config loader #1680
  • tls: allow multiple key and cert parameters for RSA+ECDSA #1663
  • permit specifying haraka plugins w/o haraka-plugin- prefix #1645
    • in config/plugins and resultstore
• Improvements
  • connection.geoip replaced by haraka-plugin-geoip #1645
  • connection.asn replaced by haraka-plugin-asn #1645
  • permit specifying npm packaged plugins w/o haraka-plugin prefix #1647
  • normalized connection properties #1547, #1577
  • Rspamd: fix spambar for negative scores #1630
  • set connection.remote.is_private early
    • replace calls to net_utils with remote.is_private test
• Bug Fixes
  • Tidy-up graceful shutdown and fix for non-cluster mode #1639
  • Fix data.headers plugin crash #1641
  • Fix access plugin crash #1640
  • Minor DKIM fix #1642
  • do not set TLS timer if timeout=0 #1632
  • do not overwrite config/host_list on install #1637
  • correct smtp_forward cfg for multiple rcpts #1680
  • fix TLS timeout errors #1665

Release Page

Github Home Page for Haraka

Project Home

 

ANNOUNCE: Haraka v2.8.9

Note this release contains a major security fix for those using the attachments plugin. Previous versions of this plugin allowed remote code execution using specially crafted zip files. Users are urged to upgrade as soon as possible.

• New Features
  • Support outbound.pool_timeout of 0 to effectively disable pooling. #1561
  • Added never_add_headers option to rspamd plugin. #1562
  • rcpt_to.routes URI format w/ LMTP support #1568
• Improvements
  • The delay_deny plugin now has a whitelist mode (vs blacklist). #1564
  • Don’t show the private key in logs for dkim_sign. #1565
  • update geoip for compat with newer ES (#1622)
  • drop node 0.10 testing / official support (#1621)
  • watch plugin displays UUIDs as URL (#1624)
  • Catch errors on header decode in rfc2231 #1599
  • Attachment plugin updates (#1606)
  • add outbound.ini pool_timeout example setting #1584
• Bug Fixes
  • Fixed some small documentation issues. #1573, #1616, #1612
  • Fixed AUTH PLAIN when it spreads over two lines. #1550
  • Fixed dkim_verify calling next() too soon. #1566
  • Fixed bugs with outbound pools who shutdown before we QUIT. #1561, #1572
  • outbound issues #1615, #1603
  • Fixed adding/removing headers in rspamd plugin. #1562
  • Fixed process_title not shutting down. #1560
  • fix a spurious error emitted by p0f (#1623)
  • fix header version hiding (#1617)
  • messagestream returns destination (#1610)
  • plugins.getdenyfn now passed 3rd params arg (#1591)
  • Fix scope of spf logdebug (#1598)
  • fix rabbitmq deliveryMode bug (#1594)
  • fix dkim_sign TypeError with null mail_from.host (#1592)
  • fix dkim_sign attempting to lower an undefined (#1587)

Setting Up Haraka to send to mailchimp/sendgrid/SES etc

One really useful way to use Haraka at a business is to be a local fast cache to forward on to mailchimp/sendgrid/SES or one of the many available transactional mail senders out there. These services offer bounce management, tracking, and also manage deliverability for you, and yet often their sending systems are slow, which can affect your app. By installing Haraka locally you can keep all those benefits, but have your system return from sending mail much faster.

First, install haraka and create an instance of it:

$ sudo npm install -g Haraka
... # lots of output
$ haraka -i /var/apps/haraka

Now edit `/var/apps/haraka/config/plugins` to define what Haraka plugins will get loaded. Here’s my simple config for this setup:

# this lets you view how much stuff your haraka instance is doing
process_title
# this lets you receive inbound mail, but is mostly not required
rcpt_to.in_host_list
# this is a custom plugin I'll detail below
relay_via_external
# this lets you configure who can relay (i.e. your app)
relay
# Check mail headers are valid
data.headers
# this lets you configure where inbound mail goes (also mostly not required)
queue/smtp_forward
# this is just a safety net
max_unrecognized_commands

You’ll need an entry in the `config/host_list` file:

$ hostname > /var/apps/haraka/config/host_list

You’ll want to allow localhost to relay:

$ echo 127.0.0.1 > /var/apps/haraka/config/relay_acl_allow
$ echo ::1 >> /var/apps/haraka/config/relay_acl_allow

Copy this into your `/var/apps/haraka/config/relay.ini` file:

[relay]
acl=true
relay=all

Prevent `smtp_forward` from processing outbound mail:

$ echo 'enable_outbound=false' > /var/apps/haraka/config/smtp_forward.ini

Edit your `smtp.ini` file to make it listen on a higher port so you don’t have to listen on port 25 by setting the `listen=` line to: listen=[::0]:2525, and set the `nodes=` line to use as many CPUs as you think is reasonable for your setup.

Finally, copy this plugin into `/var/apps/haraka/plugins/relay_via_external`:

var external_smtp = {
    auth_user: '<your username here>',
    auth_pass: '<enter password here>',
    priority: 0,
    exchange: 'smtp.mailgun.org', # or whoever else
};

exports.hook_get_mx = function (next, hmail, domain) {
    // All relaying goes via external service
    return next(OK, external_smtp);
}

Remember to edit the username and password – get them from your mail service provider.

And that’s it. There are various ways of launching Haraka (see the contrib directory on github) which keep it starting at startup. But to test it you can simply launch: `haraka -c /var/apps/haraka`.

Now just make your app use localhost on port 2525 as your mail server, with no authentication necessary, and your mail will go out MUCH faster.

ANNOUNCE: Haraka v2.8.8

• Changes
  • removed UPGRADE.doc to wiki
• Improvements
  • support + wildcard in aliases plugin #1531
  • Support dkim_sign with outbound.send_email() #1512
  • spf: always check remote IP, then public IP if != pass #1528
  • spf: diplay IP used for SPF eval #1528
• Bug Fixes
  • handle missing wss section in http.ini #1542
  • fix leak on socket write error #1541
  • add results property to outbound transaction #1535
  • don’t unref unref’d wss server #1521

Announce: Haraka v2.8.5

I haven’t bothered posting about the earlier 2.8.x releases (beyond 2.8.0) because they have mostly fixed small bugs that we introduced in the 2.8.0 release. This is the first release to add significantly new features.

• Changes
  • The connection object is now passed to get_plain_passwd. Older modules should continue to work as-is.
  • The reseed_rng plugin now just uses the Crypto module from core. Though it seems this plugin should be irrelevant with newer versions of node.js
• New Features
  • Outbound mail now uses pooled connections, only sending a QUIT message if the connection has been idle for a while.
• Improvements
  • Shut down and reload (via haraka -c <path> --graceful) is now graceful – allowing current connections to finish and plugins to clean up before ending.
• Bug Fixes
  • Bind maxmind version to ignore API change (#1492)
  • Fix encodings when banners are used (#1477)
  • Various DKIM fixes (#1495)

Upgrading should be fairly transparent, especially for current 2.8.x users.

Happy Haraking!

ANNOUNCE: Haraka 2.8.2

Note: We shortly after this released 2.8.3 which fixes config merging, a feature we added in 2.8.0. It didn’t seem to really deserve a blog post.

This is a minor bugfix release, mostly to fix the broken config/plugins file we accidentally shipped with v2.8.0.

Changes

• Added Node v6 to travis tests

New Features

• Added bin/haraka –qunstick <domain> to flush all mails for that domain (#1460)

Improvements

• Make bin/haraka –qlist show much more information (#1452)
• Allow CIDR ranges in no_tls_hosts (#1450)

Bug Fixes

• 2.8.0 was shipped with a broken config/plugins. (#1453)
• Stop haraka dying when ldap connections fail (#1456)
• Pick up domain specific config correctly in ldap (#1456)

Announce: Haraka v2.8.0

This release represents a huge leap forwards for Haraka users everywhere. We came very close to calling this v3, but since we didn’t break any APIs we stuck with v2.8.

This release contains work from 15 contributors around the world, and we thank them all for their time and effort.

Upgrading is simply a matter of running: npm install -g Haraka. Following an upgrade to 2.8 we recommend deleting any config files in your personal Haraka config folder that you have not modified. The reason for this is that Haraka will now load non-modified config data from the core Haraka folder. This change makes future upgrades much easier.

The major new features in this release are:

• The ability to write plugins as npm packages
• The merging of config data to allow minimal config in your local config folder
• Many TLS fixes including the ability to use outbound TLS without a certificate, meaning we now enable outbound TLS by default

The full list of changes are:

• Changes
  • updated dependency versions (#1426, #1425)
  • use utf8 encoding for body filters (#1429)
  • remove spameatingmonkey from tests (#1421)
  • replace ./constants.js with haraka-constants (#1353)
  • Document HMail and TODO items (#1343)
  • Copy only a minimal config/* by default (#1341).
  • cfreader/* removed to haraka/haraka-config (#1350)
  • outbound and smtp_client honor tls.ini settings (#1350)
  • outbound TLS defaults to enabled
  • lint: remove all unused variables (#1358)
  • replace ./address.js with address-rfc2181 (#1359)
• New Features
  • smtp_forward: accepts a list of backend hosts, thanks @kgeoss (#1333)
  • config: add array[] syntax to INI files (#1345)
  • plugins.js: support require(‘./config’) in plugins
  • Load plugin config from own folder and merge (#1335)
  • Allow original email’s Subject to be included in bounce message (#1337)
  • new queue/smtp_bridge plugin, thanks @jesucarr (#1351)
• Improvements
  • early_talker: supports IP whitelisting (#1423)
  • loading plugins as packages (#1278)
  • removed TLD stuff to haraka/haraka-tld (#1301)
  • removed unused ‘require(‘redis’) in plugins/karma (#1348)
  • improved MIME header support per rfc2231 (#1344)
  • tls options can be defined for outbound and smtp_* (#1357)
  • explicitly disable SSLv2 (#1395)
  • cache STUN results
  • xclient plugin improvements (#1405)
  • tls: Set verify=NO correctly when no certificate presented (#1400)
  • improved message header decoding (#1403, #1406)
  • bounce: skip single_recipient check for relays/private_ips (#1385)
  • rspamd docs: Clarify usage of check.private_ip (#1383)
  • if rcpt_to returns DSN in msg, log it properly (#1375)
• Bug Fixes
  • fix out-of-range errors from banner insertion (#1334)
  • dkim_verify: Call next only after message_stream ended (#1330)
  • outbound: remove type check from pid match (#1322)
  • lint: enable no-shadown and remove all shadow variables (#1349)
  • spf: fix log_debug syntax (#1416)
  • auto_proxy: fix a starttls loop (#1392)
  • fcrdns: corrected err variable name (#1391)
  • rspamd: Fix undefined variable (#1396)
  • dkim_verify: Fix header handling (#1371)
  • smtp_client: fix remote_ip (#1362)

Happy Haraking!

Express, Lost Sessions, and Windows

One of the harder bugs I’ve ever had to track down.

Most of our office uses Macs. One developer alone uses Windows. He would regularly see the weirdest issue where he couldn’t log into our app – he’d try time and time again, and maybe on the 4th or 5th try it would work.

He even hacked in some delays via Javascript’s setTimeout() function to try and fix it, knowing it was a weird bug with lost sessions.

After helping him debug a problem on his machine, and seeing him go through this over and over, it was finally time to get down to the meat of this issue. Why were sessions not getting saved for Windows users, but were just fine for Mac users?

Every problem related to a call to res.redirect(). First problem located.

So why would this work fine for Mac and not Windows? It wasn’t even browser specific.

Thankfully I have a large back-story in various networking related activities. Windows and Macs have different networking stacks, and different ways they cut apart packets and so on.

I also knew quite well how Express (the Node.js HTTP server stack we use) saved the session, due to various unrelated deep dives into the code.

Express saves the session by hijacking res.end(). It turns out that when you do a res.redirect(), on Windows it will likely get the headers in a single packet, and the body in another, but perform the redirect before even seeing the body (because it’s empty and irrelevant). But res.end() isn’t called until the HTTP request is completed. This means the Windows boxes can get the redirect, request the redirected URL, and get access to an unsaved session before res.end() has time to completely save the session.

The fix? Now in our code we hijack res.redirect() to perform req.session.save() before performing the actual redirect.

This fix should probably go into express-session. I will work on a pull request.

Code for the curious (put this in some middleware):

 var redirect = res.redirect;
 res.redirect = function (path) {
 res.redirect = redirect;

 if (req.session) {
 req.session.save(function (err) {
 if (err) console.error(err);
 _finish();
 });
 }
 else {
 _finish();
 }

 function _finish () {
 if (/\&utm_/.test(path)) {
 return res.redirect(path);
 }
 if (req.query.utm_campaign && req.query.utm_medium && req.query.utm_source) {
 var extras = qs.stringify({
 utm_campaign: req.query.utm_campaign,
 utm_medium: req.query.utm_medium,
 utm_source: req.query.utm_source,
 });
 if (/\?./.test(path)) {
 path = path + '&' + extras;
 }
 else {
 path = path + '?' + extras;
 }
 }
 return res.redirect(path); 
 }
 }