News Archive
2009-03-18 Server Side Solutions is now offering an eWall + Message Sniffer package
Server Side Solutions, creator of eWall, is now offering an eWall + Message Sniffer package, as well as Message Sniffer subscriptions and renewals. Please visit the SNF section of their website for more information.
2009-03-17 Updated Windows Installer
Our Windows installer (available on the Products page) now directly supports the following platforms:
- Alligate
- IceWarp
- Imail + Declude
- Imail + MXGuard
- Imail stand alone (using MINIMI)
- MDaemon (as a plugin)
- SmarterMail + Declude
- SmarterMail + MXGuard
It also provides the option to install SNFServer and SNFClient without any particular integration in case you want to do something custom or integrate with a platform that is not yet supported by the installer.
A number of items have been removed from the products page now that they are included in the new installer. Also note that the installer is up to date and now uses the CURL based getRulebase.cmd script.
Note: There are many more Windows based platforms that are compatible with SNF. Some of those install the SNF engine as part of their own installation process. Others require some manual tweaking. Please use the custom / other option from the installer to help with those.
2009-03-17 WIN* SDK (DLL) Prelimary Release
A number of folks have recently asked us about our SDK. Although we are not yet finished with the SDK package (documentation, examples, etc...) the 32 bit DLL itself has been in production on a number of large systems for several years now.
From now on a snapshot of the SDK package will be available from our Products page so that folks can download the package, try it out, and give us feedback on how we can improve it.
Yes, for those who are about to ask, we have a 64 bit DLL also (available upon request). Once we've had some more fun testing it we will add it to the SDK package.
2009-03-17 SNF4ASSP Posted
Last year on the prompting of several SNF users we built an ASSP plugin for SNF. We held back posting it to our site because we wanted to see more testing and feedback before making it public. Now we've posted it.
2009-02-02 Announcing ClamAID - Clam AV installer for Windows
We've noticed that folks often have trouble getting Clam AV (the free open source anti-virus scanner) working correctly on their mail servers, so we've created a free product to help solve that: ClamAID (Clam AV Assisted Install Device).
What ClamAID does is collect all of the bits and pieces that make ClamAV work, configure them, install them, and get them running with your email / filtering platform.
So far ClamAID supports IceWarp, Declude/IMail, and Declude/SmarterMail. We will add support for additional platforms as requested (time permitting).
Please take a look, keep us posted on your progress, and tell your friends about ClamAID if it helps you. If you have any questions or run into problems then please let us know.
2008-10-09 SNF Now directly supported in IMGate!
Message Sniffer is now directly supported in Len Conrad's IMGate. IMGate + SNF allows you to move your spam filtering out in front of your mail server improving scalability, stability, and performance.
Here are some links:
http://www.imgate.net/?page_id=101
http://www.imgate.net/?page_id=111
2008-07-31 Installers Posted (available in the Products section)
We have re-posted our Windows Installer for the new Client/Server version of SNF 3.0. This installer will help you upgrade from previous versions of SNF if you are using any of:
- IMail + Declude
- IMail + mxGuard
- SmarterMail + Declude
Other combinations are also supported and often detected automatically including raw client/server installations for use on systems we don't know about yet ;-)
This installer is relatively new (but well tested in our lab). Please keep us posted on how it works for you.
If you prefer to get the new SNF and install it yourself:
http://www.armresearch.com/message-sniffer/download/SNFWinClientServer3.0.zip
MDaemon users-- don't forget that we nave an installer for the new MDaemon plugin also!
If you prefer to install the new SNF MDaemon plugin manually:
http://www.armresearch.com/message-sniffer/download/SNFMDPlugin.3.0.zip
Linux/BSD/OSX users-- If you haven't heard, the new *nix distribution of SNF has been updated with improved V3 specific instructions and example control & update scripts. Also - if you had trouble compiling SNF before on your PowerPC or 64bit box the latest version includes big/little endian detection and bug fixes. At this time there are no known problems on all of these platforms. As always: Keep us posted please :-)
http://www.armresearch.com/message-sniffer/download/SNFSourceClientServer.3.0.1.zip
Everyone should upgrade to the newest version as soon as practical. While we will continue to support version 2 for a time, version 2 of SNF is deprecated. Of course, that's not the only reason to upgrade. SNF Version 3.0 has many improvements that reduce leakage, reduce the chance for false positives, reduce administration costs, and reduce system loads.
- More efficient, fully multi-threaded scanning engine.
- Realtime collaborative IP reputation system.
- Realtime rulebase checking and telemetry (no need to upload logs)
- Realtime system status information in XML format.
- Integrated, customizable rulebase update mechanism.
You can find links to our latest distribution files on our Products page. You can find installation guides and upgrade notes in the documentation section.
Some of our site is still under construction (it is large). If you find something under construction that would help you please let us know and we will reschedule that work to get it done more quickly. In the mean time we'll be happy to answer your questions directly.
2008-07-31 2 Millionth Rule!
We have reached our 2 Millionth Rule! -- Our rule bots now have more than 2 Million heuristics available for activation at any moment. When new spam is spotted that matches an old rule, that rule is reactivated automatically.
The vast majority of our rules have been coded by hand over the years by our amazing Rule-Techs (The SortMonsters). These highly trained professionals work around the clock (24x7x365) and consistently produce the most accurate rules available anywhere. They are really a fantastic team and a great bunch of folks to boot. :-)
At present about 122315 rules are typically active at one time.
Our most active rule at the moment was coded some 2062 days ago (has it been that long? Wow!).
Here's to the next 2 million !
2008-7-14 New verson of eWall includes tight integration with SNF!
The newest version of eWall from Server Side Solutions includes direct support for SNF:
- Simplified installation - just provide your license ID and Authentication string.
- eWall communicates directly with the SNFServer via XCI for speed and efficiency.
- New SNF specific actions and conditions.
- Automated filter generation tools in the "New Agent Wizard".
Here is a link to the announcement: http://forum.sssolutions.net/showthread.php?p=14524
2008-07-12 Rulebase Delivery System Upgraded
Our rulebase delivery subsystem has been upgraded. The new system supports 10x the previous bandwidth and a minimum of 5x the the number of transactions per second.
2008-07-10 *nix Source Distribution Upgraded to 3.0.1
The *nix source distribution has been updated to include Version 3 specific install instructions and to correct a minor bug.
2008-06-26 It's official. SNF Version 3.0 is Ready!
Back in Q1 we were sure we'd be ready with the new SNF after nearly a year of testing on both large and small systems. What a surprise!
After publishing the first release candidate we went from version 1-5 to version 2-27 at a breathtaking pace!
Thank you to everyone who has tested, poked, prodded, and twisted the new SNF -- not to mention keeping up with all of those updates during the final phase of testing. I can't imagine getting to this point without your patience, trust, attention to detail, and persistence! Bravo!
Without further fanfare: Today the latest release candidate becomes the official production release of Message Sniffer (SNF) Version 3.0.
The changes:
- Minor updates to readme files.
- Changed the build / version information and recompiled.
- Removed redundant comments from the configuration file.
We have been bug free for more than 2 months with several hundred systems using the new engine.
You can download the latest distributions from the products page:
http://www.armresearch.com/products/index.jsp
You may also notice that we've published our new web site! There are a few bits of documentation still under construction here and there, but we're well on our way to filling those in along with a stream of continues improvements and additions based on our work with you!
Once again, Thanks to everyone for a fantastic job!
Thanks for all of your support, comments, and efforts! As always we're hear to help. Now, onward to the next upgrade... always work to do ;-)2008-06-20 ARM Research Labs Launches New Website!
2008-06-10 Final RC before Version 3 (fingers crossed)
The latest SNF distributions have just been posted:
- SNFMulti engine 2-9rc 25
- SNFClient 2-9rc 7
This release is a performance update, no new bugs in many weeks now.
Here is a snip from the change log:
20080524 - Version V2-9rc2.25.7
- Optimized networking library for additional speed & stability by moving receive buffer allocation from heap to stack (automatic).
- Optimized timing parameters in SNFClient for improved speed. Polling dealys are now reduced to 10ms from 30ms.
- Removed speed-bug in SNFClient, 100ms guard time between retries was always executed after an attempt (even a successful attempt). The guard time is now condition and only fires on unsuccessful attempts.
- Updated XCI server logic to ensure non-blocking sockets for clients in all socket implementations.
PS: ****** We expect to begin wide testing of two new pieces of software soon: Windows Installers for the MDaemon plugin and Command Line versions of the new SNF. Stay tuned!
2008-04-25 New version: Engine 24, MDPlugin 6
This release is an upgrade more than a bug fix. Replace your SNFServer.exe or snfmdplugin.dll as appropriate.
No changes have been made to the configuration file.
This version improves memory management in the SNF Engine for improved performance, improves the header injection mechanism for improved reliability, and improves logging for IP scans done with the MDaemon plugin.
As usual you can get the latest distributions here:
Here is an excerpt from the change log (this time from the MDaemon plugin change log since it contains all changes from the last version):
20080424 - Version V2-9rc6.24.6
- Refactored snfScanData.clear() to reduce heap work and fragments.
- Added mutex to scanMessageFile() entry point just in case some app attempts to put multiple threads through a single engine handler. scanMessage() is already protected and fully wraped by the new scanMessageFile() mutex.
- Added non-specific runtime exception handling to XHDR injection code.
- Added 2 retries w/ 300ms delay to remove original message in XHDR inject code. If remove fails after 3 attempts the injector throws.
- Added 2 retries w/ 300ms delay to rename temp file to msg in XHDR inject code. If rename fails after 3 attempts the injector throws.
- Added IPTest logging.
2008-04-16 New Version: Engine 23 - fix for network bug on some win* systems.
This update fixes a bug that effects some Win* systems.
Please replace your SNFServer or snfmdplugin.dll and your SNFClient.
You can always get the latest distribution here:
http://kb.armresearch.com/index.php?title=Message_Sniffer.GettingStarted.Distributions#NEW_SNF_V2-9_Wide_Beta
Here is a snippet from the change log:
20080416 - Version V2-9rc2.23.6
-
Fixed bug where SNCY open() would fail on some Win* platforms with
WSAEINVAL instead of the standard EINPROGRESS or EALREADY which were expected.
Also added WSAEWOULDBLOCK to cover other "ambiguities" in windows sockets
implementations. InProgress() on Win* now test for any of:
WSAEINPROGRESS, WSAEALREADY, WSAEWOULDBLOCK, WSAEINVAL
2008-04-13 New Version SYNC bug fix! SNFEngine 22
It seems that in our last update we introduced a bug that effects SYNC operations for some customers - particularly those with longer network transit times to our servers.
The bug could cause SYNC sessions to fail either consistently or intermittently depending on the transit time. If SYNC sessions consistently fail then the new UpdateReady feature will not fire. GBUdb collaboration is also diminished with failed SYNC sessions.
A new version has been posted that solves this problem. Please upgrade your SNFServer or snfmdplugin.dll files from the new distributions as soon as possible to avoid missing telemetry, UpdateReady information, and GBUdb collaboration traffic.
We have also included a new build of the SNFClient program since it uses the same networking library. Although it is unlikely this bug would cause a problem with the SNFClient program you should update to the newest build to be sure.
No configuration changes are necessary with this update.
Here is a description of the changes in this newest distribution:
20080413 - Version V2-9rc2.22.6
- Fixed bug in TCPHost.open() where [WSA]EALREADY was not counted as a version of [WSA]EINPROGRESS. This would cause open() to throw an unnecessary exception when a socket open() required extra time.
20080413 - Version V2-9rc2.21.6
- Extended timeout for SYNC session open() to the full session length. This way if a session takes a long time to open it still has a shot at success.
2008-04-11 Latest RC release SNFMulti 20, SNFServer 2, SNFClient 6, MDaemon 5
The newest RC release has been posted in the usual location:
There are NO changes to the configuration file. You need only replace SNFServer.exe and SNFClient.exe and/or snfmdplugin.dll (as appropriate to your system). This release resolves all known bugs / tweaks.
Snippets from the change log:
20080411 - Version V2-9rc2.20.6
- Adjusted snfNETmgr to use non-blocking open in SYNC sessions. Open timeout is 1/3 of the session timeout. Session timeout is 2 * Session pacing. Open polling uses golden spiral delay from 10ms to 340ms.
20080410 - Version V2-9rc2.19.6
- Adjusted XCI manager to use new snfCFGPacket paradigm in checkCFG().
- Adjusted snf_RulebaseHandler::addRulePanic() to use MyMutex and eliminated the AutoPanicMutex and waiting scheme.
- Refactored scanMessage() to use a ScopeMutex() rather than lock()/unlock().
- Refactored scanMessage() to use MyCFGPacket.isRulePanic() test.
- Redesigned snfCFGPacket handling to automate grab() / drop() functions.
- Fixed lock-up bug: Redesigned AutoPanic posting and checking mechanisms to eliminate potential dead-lock condition. Under some conditions a precisely timed auto-panic posting could cause the RulebaseHandler mutex and the AutoPanicMutex to become intertwined leading to a cascading deadlock. When this occurred all XCI processing threads and eventually the XCI listener thread would become blocked waiting to get the current configuration.
20080409 - Version V2-9rc2.18.6
- Enhanced XCI exception handling and logging to provide additional detail.
- Added code to explicitely check for zero length files in scanMessagFile().Previously a zero length file would cause the CBFR module of the filter chain to throw an invalid buffer exception. Now if the message file is empty scanMessageFile() will throw a FileError stating FileEmpty!.
20080407 - Version V2-9rc2.17.6
- Enhanced exception reporting in snfXCImrg
2008-04-05 New Version Engine: 16, Client 6
The newest distributions for the Command Line (Std Test Package), MDaemon plugin, and Source have been posted. You can find them
here as always:
http://kb.armresearch.com/index.php?title=Message_Sniffer.GettingStarted.Distributions
This update is important because it includes a bug fix to the networking library. This update also includes some tweaks intended to improve network performance under heavy traffic conditions.
Please upgrade to the new DLL, SNFClient, and SNFServer. There is no need to change your configuration file ;-)
A snippet from the change log:
20080405 - SNFServer V2-9rc2.16.6
- Reduced safety limits on status reports to 100K for status reports and 100K for samples. Previous values were 10M. Most full sessions from the busiest systems are < 50K total.
- Recoded sendDataTimeout() to break uploads into 512 byte chunks and insert delays only when a chunk is fragmented. This methodology improves reliability on Win* systems without any significant penalty on systems that don't need socket sends() to be in smaller chunks.
- Fixed TCPClient::transmit() and TCPHost::transmit() bug where returned byte count might be -1. Now returned byte counts can only be 0 or more.
2008-03-27 More progress SNF2-9 SNFMulti engine goes to version 15
Short version:
Here is a new beta/rc release. The changes are internal and should solve a bug that happens on a handfull of systems. You should upgrade so that you're on the latest version. If you're not having trouble you can put off upgrading until some later time (but not too long please).
Please find the newest release here:
The long version:
This release goes further to eliminate the "hanging" bug on those few systems that see it. One case should be solved completely by this revision and possibly all cases (we shall see).
This release also eliminates a minor bug (not worth a revision) that was in the previous release. It seems I failed to remove a line of code that forces the Debug mode in SNFServer before pushing out the last release -- so SNFServer in the previous release would be in debug mode no matter what -- thus creating extra monitor data on the screen (if not run as a service or piped to /dev/null).
The big change in this release is in the snfNETmgr module that handles SYNC operations (GBUdb & Telemetry). The previous version used blocking IO and a separate thread (TCPWatchdog) to kill off connections that lasted too long. The new version uses non-blocking IO and has been refactored to consolidate some of it's communications routines.
In one case the "hanging" bug presented as a loss of telemetry without errors or exceptions. It appeared from the debug data that the snfNETmgr thread had gotten stuck in an IO call and that even though the TCPWatchdog thread had killed the connection the function call never returned.
The theory supporting this change is that after some number of these TCPWatchdog events the TCP stack might become unstable on some systems and cause this kind of behavior. The new non-blocking methodology eliminates this possibility.
It is possible, if the above theory is true in any way, that this change will solve the other "hanging" cases also -- In those cases the snfXCImgr thread appears to get stuck while attempting to accept() another client. If the TCPWatchdog methodology used before did cause instability in some way to cause this, then this presentation of the "hanging" bug should also disappear.
If the new revision doesn't solve the XCI related "hanging" bug then the addition of very detailed status tracking in the snfXCImgr module should help us see more clearly where to look.
Excerpts from the change log:
20080326 - SNFServer V2-9rc2.15.4
- Refactored snfNETmgr::sync() to consolidate non-blocking io routines.
- Added detailed thread status data to XCI listener thread.
- Fixed minor bug in main (not changing revision), Debug flag for internal use was left on in the last build cycle. It is commented out now.
20080325 - SNFServer V2-9rc2.14.4
- Updated snfNETmgr with comprehensive thread status data.
- Refactored snfNETmgr::sync() to check a Timeout, removed TCPWatchdog.
20080325 - SNFServer V2-9rc2.13.4
- Upgraded TCPWatcher code to use new threading features (type, status).
2008-03-25 New 2-9rc Versions Posted Client - v2, Server - v4, MDaemon DLL v4, Engine v12
Three new releases today:
SNFv2-9rc2.12.4.StdTestPackage.zip
SNFv2-9rc4.12.4.MDaemon.zip
SNFv2-9rc2.12.4.source.zip
You can find them here as usual:
http://kb.armresearch.com/index.php?title=Message_Sniffer.GettingStarted.Distributions#NEW_SNF_V2-9_Wide_Beta
Some items of note:
There is an obscure, intermittent bug in SNFServer that has been reported on a handfull of systems. The vast majority of systems run normally (including our lab systems) for hundreds of days -- only stopping when we tell them to.
The bug manifests as either:
SNFServer stops listening for requests.
OR
SNFServer stops sending telemetry.
In both of the above cases there are no errors in the logs, no core dumps, no unhandled exceptions, no corruption of any kind--
Only two cases show any kind of pattern so far:
- In one case SNFServer will stop sending telemetry after approximately 1 day give or take a few hours. Scanning and all other functions continue normally.
- In another case SNFServer appears to stop accepting requests via XCI after approximately one week give or take a few days.
- Other cases are completely random (est fewer than 5 cases total).
If you come across this scenario please let us know all of the data you can about the situation and then please run your SNFServer in debug mode (see next item) to help us track down this critter.
SNFServer now has a debug mode. If "debug" or "Debug" are found in the path to the SNFServer.exe then debug mode is turned. Most commonly to run SNFServer in debug mode rename it to SNFDebugServer.exe.
When in debug mode SNFServer will make a thread status report to the console once per second along with the usual activity information. The idea is to pipe all of this information to a log file so that when the above bug occurs we can record the status of all of the active threads at that time, before, and after.
For example:
/SNF/SNFServer.exe /SNF/snf_engine.xml > debuglog---- now some good news ---
There is a new feature in SNFServer. When there is a new rulebase file available SNFServer can call a user-defined script to retrieve the new rulebase file. We've also provided that script and set up the default settings to call it ;-)
The script name is getRulebase.cmd on Win* systems and simply getRulebase on *nix systems. Please read the readme files and check your configuration files to make sure that the script is setup properly for your system. Wget and Gzip utilities are included in all of the above distributions for your convenience.
If the script fails to replace the rulebase file then it will be retried after 3 minutes (default). Retries will continue until the script is successful.
-- The feature can be turned off.
-- SNFServer still produces an UpdateReady.txt file so if you want to
continue using that methodolgy nothing will break -- though you should
turn off the
-- If you write your own script or want to launch the script some other way (such as calling cmd or start with special options) then you can do that -- but be careful! The update-script engine runs in it's own thread and makes a system() call when triggered. If your script fails to return then the update-script thread will be stuck waiting for it to return. Remember: what you put into the call= attribute will be passed to system() when the feature is triggered. The best way to do anything special there is to write a script that does what you want and have the update-script mechanism call that script. It's probably not a good idea to put a lot of special switches and "other craziness" in the call= attribute --- If you need them, put them in your script and keep the call= simple :-)
2008-03-20 MDaemon Plugin SNFv2-9rc4.11.4 Posted
I have just posted the latest beta (release candidate) MDaemon plugin.
You can find the latest betas here:
This distribution includes an automated update utility that is triggered from the SNFServer engine running in the plugin. When a newer rulebase file is available an UpdateAvailable.txt file is created in the SNF directory. The getRulebase.cmd script can be scheduled to run once per minute. When the UpdateAvailable.txt file is present the script will download, validate, and install the latest rulebase file. Before using the getRulebase.cmd script be sure to edit the top of it to establish the correct working directory, license ID, and authentication string.
Engine improvements and updates to the SNFClient utility are also included...
A few excerpts from the change log:
20080319 - Version SNF2-9rc4.11
- Added IPScan on-off to snfmdplugin.xml. This allows users to turn off the IPScan feature without editing the Plugins.dat file as was previously required. The feature can now be enabled or disabled at will by editing the configuration file.
- Added Configuration editor options to snfmdplugin.xml. Previously the built- in configuration function was hard coded to start notepad with the config file. Now the system() call made by the ConfigFunc() can be edited in the configuration file. The configuration file name can be appended to the command optionally. The default is still to start notepad and append the configuration file path so that it is loaded automatically. It is hoped that GUI based configuration editors for the SNF plugin will be built by third parties and in the mean time folks can now configure their favorite XML file editor to modify their SNF plugin configuration.
- Modified API use fixed shutdown bug - The plugin used to initialize the SNF scanning engine when the DLL was loaded and would shut it down when the DLL was unloaded. Now the Startup and Shutdown functions in the MDaemon plugin API. This ensures that the engine components are started and shutdown in the proper sequence.
- Included new SNFEngine core (excerpts from that change log included).
20080318 - SNF2-9rc1.11.exe Consolidated several mods/fixes
- Corrected scan error logging bug. Was posting <s/> now posts <e/>.
- Updated scan error logging to be more uniform with non-scan errors.
- Enhanced error and exception reporting in SNFMulti.cpp scanMessageFile().
- Enhanced exception handling in networking module. All exceptions now throw descriptive runtime_error exceptions.
2008-03-07 Version 2-9rc1.8.2 Release Candidate (Std Test Package) Released
This is the first release candidate for what will become version 3 this quarter!
You can find the latest updates here as they arrive:
Over the next few days we will be updating the MDaemon DLL with the new engine and a new feature or two. Then we will update the source distribution for *nix & OEM systems. Then we will be launching two SDKs -- one is a .SO for *nix systems and the other is a DLL for Win* systems. Along the way we will be launching a new web site with documentation for the new version. Then later this year (Q2 - Q3 perhaps) we'll be launching DNS based IP reputation services.
For now -- back to this moment in time and the new SNFServer and SNFClient release. There are extensive updates to both the client and server programs. Be sure to go through the readme files if you are upgrading.
Also - if you are upgrading you will want to update your snf_engine.xml file to cover the new features. (GHASP! What if I forget to do that?!!) -- If you don't get to it right away then your existing snf_engine.xml file will work fine... but do get the update process on your to-do list so you can take advantage of the new features and improved default settings.
Here is a chunk of the change log to show you what is new since version 2-9b1.5.1:
20080306 - SNF2-9rc1.8.exe (FIRST RELEASE CANDIDATE for VERSION 3!)
- Added Drilldown Header Directive Functions - When the candidate source IP comes from a header matching a drilldown directive the IP is marked "Ignore" in GBUdb and the candidate is no longer eligible to be the source for that message. This allows SNF to follow the trusted chain of devices (by IP) down to the actual source of the message. It is handy for ignoring net blocks because it can match partial IPs but it is designed to allow SNF to learn it's way through the servers at large ISPs so that the original source for each message can be evaluated directly.
- Added Source Header Directive Functions - This feature allows SNF to acquire the source IP for a message from a specific header rather than searching through the Received headers in the message. This is useful when the original source for a message is not represented in Received headers. For example: Hotmail places the originating source IP in a special header and does not provide a Received header for that IP. This feature is protected from abuse by a "Context" feature which only activates the source header directive when specific content is found in a specific received header. Using the above example, this feature can be configured so that a Hotmail source header would only be read if the top Received header contained "hotmail.com [" indicating that the ptr lookup for the header matched the hotmail domain. Note: When a source is pulled from a header directive that source is put into a synthetic Received header and injected into the scanning stream (not the message) as the first Received header.
- Added forced source IP to XCI - It is now possible to "inject" or "force" the source IP for any message by providing that IP in the XCI request or directly in a scan...() function call. This allows the calling application to provide the source IP for a message ahead of any Received headers that might be in the message. This is useful when the calling application knows the original source IP for the message but that IP is not represented in the Received headers and it is not desireable to use the Source Header Directive mechanism.
- Added forced source IP mode to SNFClient - It is now possible to call the
SNFClient utility with an IP4Address using the syntax:
SNFClient -source=12.34.56.78
The -source mode of SNFClient exercises the forced source IP feature in the XCI (see above) - Added Status Report features to SNFClient and XCI - It is now possible to
request the latest status.second, status.minute, or status.hour data via
the XCI and SNFClient. The syntax for requesting a status report using the
SNFClient is:
SNFClient -status.second
SNFClient -status.minute
SNFClient -status.hour
In addition to providing status reports the SNFClient in this mode will return a nonzero value (usually 99) if it is unable to get a status report from SNFServer. This feature can be used to verify that SNFServer is up and responding. If SNFServer is OK then the result code returned is 0. - Added result codes to SNFClient - test and XCI IP test functions - The XCI engine has been upgraded to provide the range value for the IP under test as well as the symbolic result code associated with that range. This allows the -test function to provide results that are consistent with the GBUdb configuration without additional processing: For example, if the IP falls in the Caution range then the Caution result code will be returned just as if a message had been scanned with the same IP and no pattern match occurred. The same is true for Truncate and Black range hits.
- Added Timestamp and Command Line Parameter data to SNFClient.exe.err - When an error occurs with SNFClient that may not appear in the SNFServer logs an entry is appended to the SNFClient.exe.err file. That in itself is not new. The new feature is that the entries added to the SNFClient.exe.err file now include timestamp and command line data to aid in debugging.
- Added BIG-ENDIAN Conversion - When the SNFServer program is compiled on a
system that uses a BIG-ENDIAN processor (such as a power-mac) the rulebase
load process now includes a routine to convert the token matrix from it's
native LITTLE-ENDIAN format to a BIG-ENDIAN format. This solves a bug where
Power-Mac (and presumably other BIG-ENDIAN systems) could compile and run
the SNF* software but were unable to capture spam because the token matrix
in the rulebase file was misinterpreted.
Note: The BIG-ENDIAN Conversion feature is still considered experimental because it has not yet been thoroughly tested. - Updated the Configuration Log to include all of the current configuration features and to improve it's readability.
20080207 - SNF2-9b1.7.exe
- SYNC Timeout now 2x SYNC Schedule
- SNFServer now produces an UpdateReady.txt file when the UTC timestamp on the SYNC server is newer than the UTC timestamp of the active rulebase. It is presumed that a suitable update script or program will run periodically and download a fresh rulebase file if the UpdateReady.txt file is present. The update script should remove the UpdateReady.txt file when it completes a successful download of the new rulebase file.
- Added available rulebase UTC in status reports <udate utc.../>
- Added Automatic path fixup for ending / or \
- Added option to use local time in log rotation <rotation localtime='no'/> The default is still utc.
2008-03-05 MX Uptime adds Fully Integrated SNF!
The newest version of Message Sniffer is now an integral component of MX Uptime's plugin for MailEnable. Anyone wishing to use SNF only needs to enter their license ID and authentication string, then check the box :-) Screenshot for integration.
2007-10-17 Message Sniffer Version 2-9b1.5 Wide Beta
This version is considered stable for production environments. The next release will include some minor feature additions and improved default settings (thus our long wait while we monitor installed systems and refine our data). If there are no problems with the next release then we will freeze all features and create the official production release in Q1.
2007-10-05 Message Sniffer Version 2-9b1.1 Wide Beta
At your earliest convenience, please follow the following link to read about the newest version of Message Sniffer which has just been released for wide beta testing.
The command line client/server version is available now. It is a drop-in replacement for folks who have been running the current command line version (2-3.5) with a persistent instance on Winx platforms. The version in the posted distribution file requires a P3 or better.
MDaemon and *nix (source) distributions will be coming shortly.
This new engine has been in testing on a number of production systems from the very big to the very small for quite some time. There are no known bugs at this time. None the less, please be careful :-) and read carefully!
A GREAT BIG THANK-YOU goes out to the folks who have helped us alpha test and refine this version over the previous months and weeks through scores of alpha iterations! We really appreciate the help.
Over the next few days/weeks we will be adding documentation and answering questions to help folks explore and make the most use of the new features. We will also be looking for any last minute tweaks that might be needed; and we will be building a list of any additional features and/or refinements that come to light so we can get them into the production release, or at the very least the .1 that will follow.
As always, your comments, questions, and feedback will help guide our efforts. The value of the discussions we share both privately and on this list cannot be overstated.
Thanks for your patience, trust, and participation!
2007-06-26 Rulebase Compiler Upgrade
We have just completed an upgrade to the rulebase compiler software. The new version is 20-50% more efficient - as a result, updates will be produced a bit more quickly and consistently.
There is no need to make any changes on your systems.
2007-02-05 SurgeMail adds a feature to call Message Sniffer.
2007-01-05 Rulebase Update Rate Increased by 16.6%.
Now that the new delivery server is in place and functioning properly, we have re-tuned the rulebase compilers to deliver updated rulebase files 16.6% more quickly on average.
This means that you will receive updated rules more frequently throughout the day and as a result you should also see less leakage and quicker responses to new mutations of spam.
2007-01-05 FTP Access to Rulebases Being Deprecated
Note that FTP downloads of SNF rulebases is deprecated. If you are using FTP to download your rulebase files you should switch to using http w/ gzip as soon as practical.
FTP access to SNF rulebase files will continue for a time but support may be removed without notice in the future. It's a safe bet that FTP access for SNF rulebase files will remain functional through the end of this month however.
2007-01-03 Upgrading SNF rulebase delivery servers
Over the next few days we will be upgrading the SNF rulebase delivery servers. If all goes well - nobody will notice except that downloads will become faster and (likely) more frequent.
On the off- chance that this might effect you or that something unpredicted might happen we am making this announcement :-)
Expect to see the IP change for http://www.sortmonster.net. If you have closed your firewall to outgoing traffic then this may effect you - you will need to make a new "hole". Please also note that the authentication realm has changed on our delivery servers. The old realm was "SortMonster". The new realm is "SNF".
It is possible that you may miss one or more updates during the transition. We will do what we can to minimize this possibility.
2006-10-23 Version 2-3.5 Release -- Faster Engine
The plan was to hold off until the next major release, however in light of recent increases in spam traffic we are pushing out a new version with our faster engine included. All other upgrades are will wait for the major release ;-)
The scanning engine upgrade results in a 2x speed increase that hopefully will help with the higher volumes we are seeing now. Version 2-3.5 also rolls up 2-3.2i1 which included the timing and file locking upgrades. Version 2-3.5 can be found in our wiki, in the Distributions area.
2006-06-19 Rulebase Pacing Updated
We have just reduced our rulebase update pacing from 150 minutes to 120 minutes. This means rulebase updates will now arrive 20% faster.
If you are using a scheduled task to retrieve your updates, please adjust your timing appropriately (about every 60 minutes should be reasonable provided your script checks for an updated file before performing the download).
If you are triggering your updates based on the arrival of our update notification messages then you need not take any additional action - the change will be automatic.
2006-06-07 WeightGate Available
This program is distributed AS-IS, with no warranty of any kind. You are welcome to use this program on your own systems or those that you directly support. Please do not redistribute this program except as noted above, however feel free to recommend this program to others if you wish and direct them to our wiki where they can download it for themselves. Thanks!
This program is most commonly used to control the activation of external test programs from within Declude based on the weight that has been calculated thus far for a given message.
For more information and to get WeightGate, please visit the Tools page (in the Technical Details section) in the wiki.
2006-05-12 Compressed Log Files Now Accepted!
We are now able to accept compressed log files. Compressed log files can either be in zip or gzip form. For complete guidelines on submitting compressed log files please visit the Log Files Technical Details page in our wiki.
2006-04-26 Update Notification FROM Address Changing
We are changing the rulebase update notification's FROM address to:
You shouldn't have to take any action on this, but just in case you have any filtering or whitelists set up you should change them.
2006-04-05 SNFRV2R3i1 - ready for testing...
The first in a long line of coming updates has been posted for those brave souls who wish to test or may have use for the changes. You are looking for the file: snfrv2r3i1-EngineOnly.zip You can find the current interim release, Version 2-3.2i1 (Engine Only) on the following page:
http://kb.armresearch.com/index.php?title= Message_Sniffer.GettingStarted.Distributions
Be aware - this distribution only contains the SNF executable for Winx systems and source code for BSD, Linux, & other GNU (g++) capable *nix boxen.
BTW: The source now contains a handy make file for a change ;-) Also, we are now using all gnu compilers for testing and development. We previously used Code Warrior for Winx and g++ for *nix. We now use minGW (Code::Blocks) on Winx and g++ on Linux (RHES3) for testing and development.
This release addresses two key areas that are related:
* The timing functions have been replaced using a new cross-platform Timing Module. If you are curious or interested in cross-platform development in C++ you can find more info on that module here:
http://www.microneil.com/OpenLibrary/index.html
The Timing Module simplifies a number of critical timing features in SNF and made it simple to correct some unusual timing and control conditions that would occur on some systems under very specific circumstances -- these were odd, difficult to reproduce bugs which by all indications have been solved now. That is to say, those that I have been able to reproduce have been repaired and tested -- those that I had strong theories about have also been addressed and are very likely solved -- I will know more after your reports ;-)
* During the refit I also did some additional testing and tuning to improve SNF's command-line scanner performance under heavy loads, in transition (dynamic loads) and during live configuration changes (switching from persistent mode to peer-server mode and back), and on systems with multiple processors and higher speed processors (it still works great on slower boxen too). Comparative testing in the lab shows some noticeable improvements in throughput and resilience - YMMV, I look forward to your reports.
There is NO NEED to upgrade to this version at this time unless you are looking for a tiny bit more speed or solving one of the previous timing and/or control bugs (reload, rotate & stop commands for example, or the "Adjusted Persistence Race Condition" on some bsd or linux boxes -- these are now fixed and tested as far as we can test them).
The other reason you might try the new version is if you would like to help us (and others who are cautious of early adoption) by testing the latest and greatest.
Folks using the MDaemon plugin are not effected by these updates since they apply almost exclusively to command line coordination code -- the plugin has no such code ;-) Folks using other plugins, DLLs, SNFMulti or other custom configurations are also not effected by these updates.
Please keep us posted on your results.
2006-03-10 New RuleBot F002 Online
This rulebot captures and creates geocities web links from the "chatty" campaigns. This is largely a time saver for us humans... we will focus our attention more on abstracts for these campaigns now that F002 will be capturing the raw links. Rules from F002 will produce a 60 result code (Ungrouped).
2006-03-06 New Rulebase Compliers Online
Work has been completed to upgrade the rulebase compiler bots.They are now significantly more efficient. As a result, you will be seeing updates more frequently. Previous lag was between 40-120 minutes. Current lag (sustained) is < 5 minutes. More timely updates should equate to lower spam leakage for new spam.
2006-03-06 New Rulebot F001 is Online
Rulebot F001 creates IP rules for sources that consistently failmany tests while also reaching the cleanest of our spamtraps. The rules will appear in group 63. Expect an increase in your rulebase size while F001 catches up with current spamtrap data.
2006-02-15 Updated Expired Rulebase Cleanup Code
New code has been added to the server that delivers rulebase files. The code removes any rulebase file where the license is disabled. This was a task tha was done manually, but is now automated.
If you get a 404 when you attempt to download your rulebase file then it is very likely you need to renew. If you want to check first, feel free to send us a note at support@armresearch.com.
2005-12-21 Sniffer Engine Updates
Increased Updates per Day: Standard rulebase delivery pacing has been changed from 200 to 150. This means that, on average, rulebase files will be recompiled every 2.5 hours or so. This timing will be variable based on system loads etc, but it is a significant improvement. We have sped up our rulebase delivery process by 267%!! (from 3.6 updates/day to 9.6 updates/day).
Improved IP Rule Coding: A new piece of optimization code was added to drop any Received IP rule that has 0 rule strength and is more than 30 days old. This will help to reduce false positives caused by IP rules that "hang on" after the infection/problem with the source is fixed. It also reduces the compiler workload a bit by reducing the core rulebase size.
2005-11-02 Rule Strength Analysis Upgrades
The Rule Strength Analysis upgrade makes the rule strength calculation more sensitive to the recent activity of any given rule. This will also cause rule fitness decisions to be more competitive so that the most effective rules will be more strongly selected over time.
This will improve SNFs performance in two ways:
1. Rulebase files will be smaller and will require less bandwidth to download and to load during operation. There will also be a measurable increase in scanning speed (though this is already measured in small numbers of milliseconds on most systems).
2. The smaller, more efficient files can be compiled and delivered more quickly which will allow us to increase the rate at which we deliver updates.
2005-08-11 Message Sniffer and Assert! Used to Halt New Bagle Variant
Assert! and Message Sniffer rules were quickly updated upon news that a Bagle variant outbreak had reached very high numbers according to AppRiver, a leading anti-spam service provider. Within hours customers were protected from the rapidly spreading variant, contained in compressed .RAR and .ZIP files. Though Message Sniffer primarily focuses on anti-spam content filtering, the engine can also help prevent email-borne virus outbreaks.
2005-08-01 ARM Research Releases "Assert! Message Sniffer for SMTP and Exchange"
Assert! version 1.1 encapsulates the raw power of the Message Sniffer engine with an easy, intuitive interface for Exchange or the IIS SMTP Service. Assert! is a powerful anti-spam tool that does not require a bloated feature set or period of tuning to be effective. Assert! includes a one-year subscription to the Message Sniffer spam database, which is automatically updated multiple times daily for pinpoint accuracy.
2005-07-01 AppRiver LLC and MicroNeil Research Corporation form ARM Research Labs (ARM).
With the goal of exploring ideas and raw data as a means for producing internet-based technology products, a leading anti-spam service provider AppRiver LLC and software research innovator Microneil Research Corporation have joined efforts as ARM Research Labs LLC. ARM is dedicated to strengthening the world of computing online innovations in areas such as application development, security services and other web-based operations.