canit-domain-pro administration guide

CanIt-Domain-PRO Administration Guidefor Version 10.2.0

Roaring Penguin Software Inc.13 February 2018

2

CanIt-Domain-PRO — Roaring Penguin Software Inc.

Contents

1 Introduction 191.1 Principles of Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

1.2 Handling False-Positives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

1.2.1 Spam-Control Delegation . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

1.3 Organization of this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

1.4 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2 Operation 272.1 Principles of Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2.2 Interaction between Allow Rules and Block Rules . . . . . . . . . . . . . . . . . . . 28

2.2.1 RCPT TO: Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.2.2 Post-DATA Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.3 Streaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.4 How Addresses are Streamed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.5 How Streaming Methods are Chosen . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.6 Status of Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

2.6.1 Secondary MX Relays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

2.7 The Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

2.8 Remailing Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3 Realms 393.1 Introduction to Realms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.2 Realm Names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.2.1 The base Realm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.3 Creating Realms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.4 Realm Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.5 Determining the Realm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

CanIt-Domain-PRO — Roaring Penguin Software Inc. 3

4 CONTENTS

3.5.1 Mapping an Address to a Realm . . . . . . . . . . . . . . . . . . . . . . . . 42

3.5.2 Mapping a Login Name to a Realm . . . . . . . . . . . . . . . . . . . . . . 42

3.6 Realm Expiry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.6.1 Suspending Service to a Realm . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.7 Realm Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.8 Realm Custom Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4 Streams 474.1 Introduction to Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.2 Realms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.3 The Definition of a Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.4 Users and E-Mail Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.5 Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.6 The Home Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.7 The “default” Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

5 CanIt-Domain-PRO Setup 535.1 Accessing The Web Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5.1.1 License Key Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5.1.2 Login Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

5.2 The Setup Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

5.3 Wizards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

5.3.1 Basic Setup Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

5.3.2 RPTN Setup Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

5.3.3 Dictionary Attack Detection Wizard . . . . . . . . . . . . . . . . . . . . . . 56

5.4 Verification Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

5.4.1 Wildcard Verification Server . . . . . . . . . . . . . . . . . . . . . . . . . . 59

5.4.2 SRS and Verification Servers . . . . . . . . . . . . . . . . . . . . . . . . . . 60

5.5 Mail Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

5.5.1 Outbound Relaying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

5.5.2 Outbound Relaying for Select Domains . . . . . . . . . . . . . . . . . . . . 62

5.6 Cluster Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5.6.1 Bandwidth Optimization for Copying Files . . . . . . . . . . . . . . . . . . 64

5.6.2 Altering Services on a Cluster Member . . . . . . . . . . . . . . . . . . . . 64

5.6.3 Renaming of Cluster Members . . . . . . . . . . . . . . . . . . . . . . . . . 65

5.7 Known Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65


CONTENTS 5

5.7.1 Associating Domains with Known Networks . . . . . . . . . . . . . . . . . 68

5.7.2 Overlapping Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

5.7.3 The SMTP-AUTH Pseudo-Network . . . . . . . . . . . . . . . . . . . . . . 69

5.8 Rate-Limiting Outbound Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

5.8.1 Rate-Limiting by IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . 71

5.8.2 Fine-Grained Rate-Limiting Rules . . . . . . . . . . . . . . . . . . . . . . . 71

5.8.3 Notes about Rate-Limiting Rules . . . . . . . . . . . . . . . . . . . . . . . 73

5.9 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

5.9.1 Direct Queue Injection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

5.10 System Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

5.11 Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

5.12 Theme Customization and Branding . . . . . . . . . . . . . . . . . . . . . . . . . . 79

5.12.1 Creating or Editing a Customization . . . . . . . . . . . . . . . . . . . . . . 80

5.12.2 Emergency Recovery from Bad Theme Customization . . . . . . . . . . . . 81

5.13 HTTPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

5.14 The Domain Mapping Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

5.15 The Address Mapping Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

5.15.1 Wild-Card Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

5.16 The default Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

5.17 Mapping Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

5.17.1 Central Scanning with Opt-Out . . . . . . . . . . . . . . . . . . . . . . . . 85

5.17.2 Single Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

5.17.3 Single Domain with Aliases and Mailing Lists . . . . . . . . . . . . . . . . 86

5.18 Pausing Delivery to Selected Domains . . . . . . . . . . . . . . . . . . . . . . . . . 86

5.18.1 Pausing Delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

5.18.2 Resuming Delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

5.19 The Domain Overview Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

5.20 Autotask® Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

5.20.1 Preparing Autotask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

5.20.2 Preparing CanIt-Domain-PRO . . . . . . . . . . . . . . . . . . . . . . . . . 90

5.20.3 Testing the Autotask Integration Settings . . . . . . . . . . . . . . . . . . . 93

5.20.4 Autotask Settings and Inheritance . . . . . . . . . . . . . . . . . . . . . . . 95

5.21 ConnectWise® Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

5.21.1 Preparing ConnectWise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

5.21.2 Preparing CanIt-Domain-PRO . . . . . . . . . . . . . . . . . . . . . . . . . 103


6 CONTENTS

6 CanIt-Domain-PRO Administration 1056.1 Global Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

6.2 SRS (Sender Rewriting Scheme) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

6.3 Real-Time DNS Blocklists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

6.3.1 Entering the Master List of DNS RBLs . . . . . . . . . . . . . . . . . . . . 110

6.3.2 combined.bl.rptn.ca . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

6.4 Phishing URLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

6.4.1 Malicious URL Votes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

6.4.2 Known Phishing URLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

6.4.3 Delaying Messages because of local Phishing Votes . . . . . . . . . . . . . . 115

6.5 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

6.5.1 User Privileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

6.5.2 Adding a User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

6.5.3 Editing a User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

6.5.4 Deleting a User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

6.5.5 Granting Access to Streams . . . . . . . . . . . . . . . . . . . . . . . . . . 119

6.5.6 Switching Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

6.6 Permitting Users to Opt In . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

6.7 Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

6.7.1 Creating, Deleting and Editing Groups . . . . . . . . . . . . . . . . . . . . . 122

6.8 Viewing Active Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

6.8.1 Definition of an Active Stream . . . . . . . . . . . . . . . . . . . . . . . . . 124

6.8.2 The Active Stream Display . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

6.8.3 Deleting a Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

6.9 Filtering Outbound Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

6.9.1 DKIM-Signing Outbound Mail . . . . . . . . . . . . . . . . . . . . . . . . 125

6.10 Copying Rules from One Stream to Another . . . . . . . . . . . . . . . . . . . . . . 129

6.11 Secondary MX Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

6.12 Avoiding Backscatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

6.13 Test Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

6.13.1 The PhishingAddress Plugin . . . . . . . . . . . . . . . . . . . . . . . . . . 132

6.13.2 The PhishingURL Plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

6.13.3 The OfficeMacros Plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

6.13.4 The OfficeMacro* Open Plugins . . . . . . . . . . . . . . . . . . . . . . . . 132

6.13.5 The Shortener404 Plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133


CONTENTS 7

6.13.6 The NewlySeenDomain Plugin . . . . . . . . . . . . . . . . . . . . . . . . . 133

6.14 Emergency Blocking of Delivery Status Notifications . . . . . . . . . . . . . . . . . 133

6.15 Removing All Rules and Settings from a Stream . . . . . . . . . . . . . . . . . . . . 134

6.16 Provisioning Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

6.16.1 Computer-Readable Provisioning Information . . . . . . . . . . . . . . . . . 136

7 External Authentication 1377.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

7.2 User Lookups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

7.2.1 IMAP and POP3 Authentication . . . . . . . . . . . . . . . . . . . . . . . . 139

7.2.2 LDAP Authentication and Streaming . . . . . . . . . . . . . . . . . . . . . 141

7.2.3 Azure Active Directory Streaming . . . . . . . . . . . . . . . . . . . . . . . 145

7.2.4 Program Authentication and Streaming . . . . . . . . . . . . . . . . . . . . 153

7.2.5 Program Authentication (Legacy Method) . . . . . . . . . . . . . . . . . . . 157

7.2.6 The account-info Script . . . . . . . . . . . . . . . . . . . . . . . . . . 157

7.2.7 The Rewrite User Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

7.3 Authentication Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

8 Bayesian Filtering 1618.1 Introduction to Bayesian Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

8.2 Unauthenticated Voting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

8.3 The Bayes Journal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

8.4 Site-Wide and Realm-Wide Bayes Training . . . . . . . . . . . . . . . . . . . . . . 162

8.5 RPTN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

8.6 Ruleset and Geolocation Data Updates . . . . . . . . . . . . . . . . . . . . . . . . . 163

9 Permissions 1659.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

9.2 Stream Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

9.3 Determining Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

9.4 Granting Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

9.4.1 Granting Stream Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . 167

9.4.2 Granting User Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . 169

9.5 Permission Grantability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

9.5.1 Grantability Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

10 Streams, Inheritance and the Simple GUI 173


8 CONTENTS

10.1 Simplification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

10.2 Stream Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

10.3 Special Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

10.3.1 Final Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

10.3.2 Creating Special Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

10.3.3 Deleting Special Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

10.4 The Simplified GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

10.5 Inheritance from Non-Final Streams . . . . . . . . . . . . . . . . . . . . . . . . . . 177

10.6 Inheritance from Opted-Out Streams . . . . . . . . . . . . . . . . . . . . . . . . . . 177

11 Periodic Reports 17911.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

11.1.1 Periodic Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

11.1.2 Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

11.2 Creating Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

11.3 Creating Periodic Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

11.4 Editing Periodic Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

11.5 Running a Report on Demand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

12 Locked Addresses 18512.1 Introduction to Locked Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

12.2 Preparing to use Locked Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

12.2.1 Create a new domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

12.2.2 Configure mail for the new domain . . . . . . . . . . . . . . . . . . . . . . 185

12.2.3 Inform CanIt-Domain-PRO about the locked address domain . . . . . . . . . 186

12.2.4 Associate each login name with an e-mail address . . . . . . . . . . . . . . . 186

13 Attachment Handling 18713.1 General Filename and MIME Type Rules . . . . . . . . . . . . . . . . . . . . . . . 187

13.2 Delaying Attachments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

13.2.1 Configuring the Time Delay . . . . . . . . . . . . . . . . . . . . . . . . . . 187

13.2.2 Creating Delay Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

13.2.3 How It Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

13.3 Stripping Attachments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

13.3.1 Approving the Release of Stripped Attachments . . . . . . . . . . . . . . . . 190

14 URL Proxying 191


CONTENTS 9

14.1 Configuring URL Proxying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

14.2 Proxying Known Phishing URLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

14.2.1 Known Phishing Test Point . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

15 SMTP Server Testing 19515.1 An SMTP Primer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

15.2 Testing an SMTP Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

15.3 SMTP Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

16 CanIt Storage Manager 20116.1 Storage Manager Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

16.1.1 Principles of Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

16.2 Configuring the Storage Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

16.2.1 Enabling the Storage Manager . . . . . . . . . . . . . . . . . . . . . . . . . 203

16.2.2 The Configuration Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

16.2.3 Local Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

16.2.4 Starting the Storage Manager . . . . . . . . . . . . . . . . . . . . . . . . . . 205

16.2.5 Data Stored in the Storage Manager . . . . . . . . . . . . . . . . . . . . . . 206

16.3 Backup Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

16.4 Running multiple Storage Managers . . . . . . . . . . . . . . . . . . . . . . . . . . 206

16.5 ps Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207

17 Searching Logs 20917.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

17.2 Log Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

17.3 Searching the Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

17.3.1 Performing a Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

17.3.2 Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

17.3.3 Creating a Log Search Query . . . . . . . . . . . . . . . . . . . . . . . . . . 213

17.4 Saving Log Searches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

17.4.1 Managing Saved Log Searches . . . . . . . . . . . . . . . . . . . . . . . . . 213

17.5 Log Search Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

17.5.1 Detailed Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

17.5.2 Downloading Log Lines . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

17.6 Forwarding Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

17.6.1 Enabling Log-Forwarding . . . . . . . . . . . . . . . . . . . . . . . . . . . 216


10 CONTENTS

17.6.2 Configuring Log-Forwarding . . . . . . . . . . . . . . . . . . . . . . . . . . 216

18 Tips 21918.1 Greylisting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219

18.2 Don’t Trust Sender Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

18.3 Don’t Trust Sender Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

18.4 You May Trust Relay Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

18.5 Custom Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

18.5.1 General Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

18.5.2 Things to avoid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

18.6 Group High-Scoring Messages Together . . . . . . . . . . . . . . . . . . . . . . . . 221

18.7 Roaring Penguin Best-Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

18.8 General Anti-Spam Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

18.8.1 Use Receive-Only Addresses on your Web Site . . . . . . . . . . . . . . . . 222

18.8.2 Do Not Reply to Spam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

19 Security 22319.1 Don’t Run as Root . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

19.2 Ownership and Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

19.3 SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

19.4 PostgreSQL Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

19.5 PHP Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

19.6 Network Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

19.7 Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

A The Domain Configuration Wizard 227A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

A.2 Entering the Domain Name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

A.3 Picking a Realm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

A.4 Configuring Streaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

A.5 Configuring Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229

A.6 Configuring Routing and Verification . . . . . . . . . . . . . . . . . . . . . . . . . . 230

A.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

B Release Notes 233

C A Testing Topology for CanIt-Domain-PRO 327


CONTENTS 11

C.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327

C.2 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327

C.3 Network Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327

C.4 Build the CanIt-Domain-PRO Server . . . . . . . . . . . . . . . . . . . . . . . . . . 328

C.5 Configure the CanIt-Domain-PRO Server to Relay Mail . . . . . . . . . . . . . . . . 328

C.5.1 Enable Relaying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329

C.5.2 Configure Forwarding Relays . . . . . . . . . . . . . . . . . . . . . . . . . 329

C.5.3 Rebuild Sendmail Databases . . . . . . . . . . . . . . . . . . . . . . . . . . 329

C.6 Route Test Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329

C.6.1 Direct Injection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

C.6.2 Create a Test Subdomain . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

C.7 Route Real Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

C.8 Outgoing Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331

D CanIt-Domain-PRO Architecture 333D.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333

D.2 CanIt-Domain-PRO Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334

D.3 Starting and Stopping CanIt-Domain-PRO . . . . . . . . . . . . . . . . . . . . . . . 335

D.4 Static Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336

D.4.1 Database Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336

D.4.2 Cron Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336

D.4.3 MIMEDefang Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

D.4.4 Filter Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339

D.4.5 Ticker Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340

D.4.6 Cluster Communication Settings . . . . . . . . . . . . . . . . . . . . . . . . 341

D.4.7 Storage Manager Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . 341

D.4.8 Maintenance Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342

D.5 Tuning CanIt-Domain-PRO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342

D.5.1 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

D.5.2 Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

D.5.3 Solaris-Specific tmpfs Note . . . . . . . . . . . . . . . . . . . . . . . . . . 343

D.5.4 CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

D.5.5 Sendmail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

D.6 Dealing with Overload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344

D.6.1 Tune CanIt-Domain-PRO and Sendmail . . . . . . . . . . . . . . . . . . . . 344


12 CONTENTS

D.6.2 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344

E CanIt-Domain-PRO HOWTOS 345E.1 Restoring a Database from a Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . 345

E.2 Firewall Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346

E.2.1 Firewall Rules: External Hosts . . . . . . . . . . . . . . . . . . . . . . . . . 346

E.2.2 Firewall Rules: Internal Hosts . . . . . . . . . . . . . . . . . . . . . . . . . 346

E.2.3 Firewall Rules: Intra-Cluster Hosts . . . . . . . . . . . . . . . . . . . . . . 347

E.3 Running Something after the Nightly Cron Job Completes . . . . . . . . . . . . . . 347

E.4 Hooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348

E.5 Migrating CanIt-Domain-PRO to a Different Machine . . . . . . . . . . . . . . . . . 348

E.5.1 CanIt-Domain-PRO Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . 349

E.5.2 Storage Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349

E.5.3 Migration Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349

E.6 Cloning a CanIt-Domain-PRO Machine . . . . . . . . . . . . . . . . . . . . . . . . 352

F Using CanIt-Domain-PRO with memcached 353F.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353

F.2 Using memcached . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353

F.2.1 Installing memcached . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353

F.2.2 Configuring memcached . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353

F.2.3 Single vs. Multiple Caches . . . . . . . . . . . . . . . . . . . . . . . . . . . 354

F.2.4 Configuring CanIt-Domain-PRO to use memcached . . . . . . . . . . . . . . 354

F.3 What is Cached . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355

G Using CanIt-Domain-PRO with PgBouncer 357G.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

G.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

G.3 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

G.3.1 Configuring userlist.txt . . . . . . . . . . . . . . . . . . . . . . . . . 358

G.3.2 Configuring pgbouncer.ini . . . . . . . . . . . . . . . . . . . . . . . . 358

G.3.3 Configuring CanIt-Domain-PRO to use PgBouncer . . . . . . . . . . . . . . 358

H CanIt-Domain-PRO Logging 361H.1 General Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361

H.2 Event Log Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362


CONTENTS 13

I SNMP Agents for CanIt-Domain-PRO 367I.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367

I.2 The SNMP Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367

I.2.1 Enabling the agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368

I.2.2 Configuring SNMPd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368

I.2.3 Agent Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368

J Additional Scripts 371J.1 reset-password.pl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371

K Bayes Database Back-Ends 373K.1 PostgreSQL Bayes Data Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373

K.2 Berkeley Database Bayes Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . 373

K.3 CDB Database Bayes Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373

K.4 Cluster Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374

K.4.1 Propagating Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374

K.5 Switching back to PostgreSQL Bayes Storage . . . . . . . . . . . . . . . . . . . . . 374

L System Check Tests 375L.1 Disabling System Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378

L.2 Anomaly Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378

L.2.1 Disabling Recipient Verification Anomaly Testing . . . . . . . . . . . . . . 379

L.2.2 More Details about Anomalies . . . . . . . . . . . . . . . . . . . . . . . . . 380

L.2.3 Suppressing Anomaly Notification Emails . . . . . . . . . . . . . . . . . . . 380

M The CanIt-Domain-PRO License 381M.1 THE CANIT DATA LICENSE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384

Index 385


14 CONTENTS


List of Figures

2.1 Flow of Mail through CanIt-Domain-PRO . . . . . . . . . . . . . . . . . . . . . . . 28

2.2 RCPT TO: Decision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.3 Post-Data Decision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.4 Address Streaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

2.5 Database Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

3.1 Administrative Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.2 Realm Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.3 Realm Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.4 Realm Hierarchy Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3.5 Realm Custom Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4.1 Streaming Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.1 License Key Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5.2 Login Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

5.3 Welcome Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

5.4 Verification Server Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

5.5 Verification Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5.6 Domain Routing Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

5.7 Domain Routing Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

5.8 Cluster Management Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5.9 Known Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

5.10 Known Network with Associated Domains . . . . . . . . . . . . . . . . . . . . . . . 68

5.11 Rate-Limiting Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

5.12 System Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

5.13 Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

5.14 Theme Customizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79


16 LIST OF FIGURES

5.15 Theme Customization Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

5.16 Domain Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

5.17 Address Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

5.18 Domain Overview Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

5.19 Autotask Product List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

5.20 Autotask Recurring Service Contract . . . . . . . . . . . . . . . . . . . . . . . . . . 90

5.21 Autotask Integration Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

5.22 Autotask Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

5.23 Autotask Contract Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

5.24 CanIt-Inbound ConnectWise Product . . . . . . . . . . . . . . . . . . . . . . . . . . 96

5.25 CanIt Product List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

5.26 Integrator Login ID Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

5.27 CanItBilling Management IT Solution Setup . . . . . . . . . . . . . . . . . . . . . . 99

5.28 CanItBilling Managed Device Integration Setup . . . . . . . . . . . . . . . . . . . . 100

5.29 Connectwise Agreement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

5.30 Connectwise Agreement Addition . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

5.31 ConnectWise Setup - Main Realm . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

5.32 ConnectWise Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

6.1 Global Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

6.2 Master RBLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

6.3 Phishing URL Votes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

6.4 Known Phishing URLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

6.5 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

6.6 Add User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

6.7 Edit User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

6.8 Granting Access to Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

6.9 Stream Opt-In Approval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

6.10 Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

6.11 Group Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

6.12 Active Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

6.13 Known Network with Associated Domains . . . . . . . . . . . . . . . . . . . . . . . 126

6.14 DKIM Key list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

6.15 Adding a DKIM Key Pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

6.16 DKIM Key Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127


LIST OF FIGURES 17

6.17 Copying Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

6.18 Test Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

6.19 Block Delivery Status Notifications Page . . . . . . . . . . . . . . . . . . . . . . . . 134

6.20 Provisioning Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

7.1 User Lookup List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

7.2 User Lookup Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

7.3 User Lookup: Method Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

7.4 IMAP/POP3 User Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

7.5 LDAP User Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

7.6 Azure Active Directory Main Screen . . . . . . . . . . . . . . . . . . . . . . . . . . 146

7.7 Azure Active Directory Application Registration . . . . . . . . . . . . . . . . . . . 147

7.8 Azure Active Directory Application Settings . . . . . . . . . . . . . . . . . . . . . . 148

7.9 Azure API Access Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

7.10 Azure Read Directory Permission . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

7.11 Azure API Key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

7.12 Azure Setup within CanIt-Domain-PRO . . . . . . . . . . . . . . . . . . . . . . . . 152

7.13 Program User Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

7.14 Authentication Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

9.1 Permissions Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

9.2 Permissions Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

9.3 Stream Permissions Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

9.4 User Permissions Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

9.5 Permission Grantability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

9.6 Grantable Permissions Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

10.1 Stream Inheritance Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

10.2 Stream Inheritance Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

10.3 Special Stream Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

10.4 Simplified Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

11.1 Periodic Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

11.2 Add Periodic Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

13.1 Delayed Attachments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

13.2 Attachment-Stripping Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189


18 LIST OF FIGURES

14.1 Redirected Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

14.2 URL Proxy Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

15.1 SMTP Session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

15.2 SMTP Server Test Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

15.3 SMTP Server Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

16.1 CanIt Storage Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

16.2 Storage Manager Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204

17.1 Log Search Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

17.2 Saved Log Searches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

17.3 Log Search Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

17.4 Log Search Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

17.5 Log Forwarding Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

A.1 Domain Configuration: Enter Domain Name . . . . . . . . . . . . . . . . . . . . . . 227

A.2 Domain Configuration: Enter Realm Name . . . . . . . . . . . . . . . . . . . . . . 228

A.3 Domain Configuration: Configuring Streaming . . . . . . . . . . . . . . . . . . . . 228

A.4 Domain Configuration: Configuring Authentication . . . . . . . . . . . . . . . . . . 229

A.5 Domain Configuration: Configuring Routing and Verification . . . . . . . . . . . . . 230

C.1 Network Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328

D.1 CanIt-Domain-PRO Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334

L.1 Anomaly Notice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378

L.2 Anomaly Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380


Chapter 1

Introduction

CanIt-Domain-PRO is server-based anti-spam software that stops spam from entering your network.This guide explains how to administer CanIt-Domain-PRO, and is intended for e-mail administrators.For installation instructions, please see the Installation Guide, and for end-user instructions, see theUser’s Guide.

1.1 Principles of Operation

CanIt-Domain-PRO uses many sophisticated rules and mechanisms to detect spam. These rules in-clude those in an open-source anti-spam package, and are very effective and broad-spectrum. OnceCanIt-Domain-PRO decides that a message is probably spam, it is held for review.

A more complete description of how CanIt-Domain-PRO operates is given in Chapter 2.

1.2 Handling False-Positives

Although CanIt-Domain-PRO’s rules for identifying spam are very accurate, no purely automated pro-cess can be 100% correct. That is why CanIt-Domain-PRO relies, in the end, on human intervention.In this way, it can guarantee that no legitimate e-mail message will ever be rejected, and you will neverlose an important e-mail because of automated scanning.

At first glance, it seems that requiring human intervention is a step backwards—spam messages againmust be reviewed by a person. In reality, CanIt-Domain-PRO still saves time and money for thefollowing reasons:

• CanIt-Domain-PRO includes many features to lower your workload. (These features are de-scribed later in this manual.) You can scan and categorize e-mail messages using CanIt-Domain-PRO much more quickly than using mail reader software.

• As time passes, you will begin recognize mailing-list traffic and other traffic that tends to befalsely flagged as spam, and tell CanIt-Domain-PRO to always allow that traffic. Over time, thisreduces the amount of human intervention required.


20 CHAPTER 1. INTRODUCTION

• If you are willing to take the risk of inappropriately rejected messages, you can configure CanIt-Domain-PRO to automatically reject very high-scoring messages.

1.2.1 Spam-Control Delegation

CanIt-Domain-PRO operates similarly to CanIt-PRO, except that it allows two levels of administrativedelegation. In CanIt-PRO, the system administrator can create separate streams. Stream owners canreview quarantined mail within their streams. Only the single system administrator can create streams.

In CanIt-Domain-PRO, however, the system administrator creates realms, each of which has its ownRealm Administrator. Realm Administrators, in turn, can create streams, each of which has a StreamOwner responsible for settings within the stream.

Settings in different streams do not affect other streams.

1.3 Organization of this Manual

This manual is divided as follows:

Chapter 1, “Introduction”, is this chapter. You should familiarize yourself with the terms in Section 1.4before proceeding.

Chapter 2, “Operation”, describes the principles behind CanIt-Domain-PRO’s operation.

Chapter 3, “Realms”, describes Realms. A Realm is a complete administrative unit in CanIt-Domain-PRO. You must read and understand this chapter before using CanIt-Domain-PRO in production.

Chapter 4, “Streams”, describes the concepts behind streaming. You must read and understand thischapter before using CanIt-Domain-PRO in production.

Chapter 5, “CanIt-Domain-PRO Setup”, describes basic setup steps you need to take to configureCanIt-Domain-PRO.

Chapter 6, “CanIt-Domain-PRO Administration”, describes tasks undertaken by the CanIt-Domain-PRO administrator.

Chapter 7, “External Authentication”, describes how to integrate CanIt-Domain-PRO with an externalauthentication mechanism (such as LDAP or POP3.)

Chapter 8, “Bayesian Filtering”, explains CanIt-Domain-PRO’s Bayesian filtering module. Bayesianfiltering uses statistical analysis and training so that CanIt-Domain-PRO “learns” to recognize spambased on user feedback.

Chapter 9, “Permissions”, describes how to control access to various parts of the CanIt-Domain-PROWeb interface.

Chapter 10, “Streams, Inheritance and the Simple GUI”, describes how the CanIt-Domain-PRO ad-ministrator can set up different groups of spam-handling settings and allow end-users to select fromone of a limited number of predetermined setups. The simplified interface is very useful if you wishto provide “canned” settings for unsophisticated users.

Chapter 12, “Locked Addresses”, describes how CanIt-Domain-PRO permits users to generate ad-dresses that they can give out to strangers, but that those strangers cannot in turn give or sell to


1.4. DEFINITIONS 21

third-parties.

Chapter 13, “Attachment Handling”, describes CanIt-Domain-PRO options for handling various at-tachments.

Chapter 14, “URL Proxying”, describes a CanIt-Domain-PRO feature that can help mitigate phishingattacks that trick users into visiting hostile web sites and entering sensitive information.

Chapter 15, “SMTP Server Testing”, describes a CanIt-Domain-PRO feature that lets you run a de-bugging SMTP session against a back-end mail server.

Chapter 17, “Searching Logs”, describes CanIt-Domain-PRO’s log-indexing and searching feature(available only on appliance builds.)

Chapter 18, “Tips”, contains guidelines for reducing the workload of the spam-control officer anddealing with spam more effectively.

Chapter 19, “Security”, contains information about CanIt-Domain-PRO security.

Appendix C, “A Testing Topology for CanIt-Domain-PRO”, gives tips on how to test CanIt-Domain-PRO before putting it into production. This appendix also contains useful information on productionnetwork topology, so if you are planning on using CanIt-Domain-PRO as a relay-only server, youshould read this appendix.

Appendix D, “CanIt-Domain-PRO Architecture”, discusses CanIt-Domain-PRO’s filter architecturein detail. It provides tips on tuning CanIt-Domain-PRO and describes the various configuration filesused by CanIt-Domain-PRO.

Appendix E, “CanIt-Domain-PRO HOWTOs”, gives short “how-to” recipes for performing commonCanIt-Domain-PRO administrative tasks, such as restoring a database from the text dump, or movingCanIt-Domain-PRO to another machine.

Appendix H, “CanIt-Domain-PRO Logging”, explains how CanIt-Domain-PRO logs statistics, warn-ing, and error messages.

Appendix J, “Additional Scripts”, describes some additional scripts bundled with CanIt-Domain-PROthat you might find useful.

1.4 Definitions

We use many terms related to Internet e-mail in this manual. Here is a definition of some of the termswe use.

Allow list A list of domains, senders or hosts whose e-mail is permitted through without spam-scanning.

API Application Programming Interface. In the context of CanIt-Domain-PRO, the API is a methodfor interacting with CanIt-Domain-PRO from a program or script.

Backscatter Unwanted DSNs (see “DSN”) caused when e-mail systems respond to faked senderaddresses.

Bayesian Analysis is a method whereby an anti-spam system keeps track of how often words appear



in spam and non-spam. Once enough statistics have been accumulated, the system can calculatethe likelihood that a new message is spam.

Blocklist A list of domains, senders or hosts that are blocked from sending e-mail.

CIDR “Classless Inter-Domain Routing”. A method for specifying an entire set of contiguous IPaddresses.

CanIt-Domain-PRO is an enhanced version of CanIt-PRO that allows two levels of delegation ofresponsibility. See the next three definitions for more details.

CanIt-PRO is an enhanced version of CanIt that allows flexible delegation of spam-control respon-sibilities rather than requiring a single spam-control officer.

CanIt is extra software built on top of MIMEDefang that provides sophisticated spam-managementfunctions.

Cron A UNIX program that runs tasks periodically.

DKIM “DomainKeys Identified Mail”. A mechanism for proving that a particular organization’sservers have relayed an email message. DKIM uses cryptographic techniques to assert that aparticular domain name is responsible for relaying the message. For more information, seehttp://www.dkim.org/.

DMARC “Domain-based Message Authentication, Reporting and Conformance”. A mechanism forallowing domain owners to specify a policy that recipients should use in response to potentially-spoofed messages from that domain. For more information, see https://dmarc.org/.

DNS “Domain Name System”. The mechanism used on the Internet to translate host names to IPaddresses and more generally, to associate various sorts of information with domain names.

DNSBL “DNS Blocklist”. A DNS-based system for checking in real-time whether or not hosts ordomains should be blocked. Sometimes referred to as “Real-time Blocklist” or RBL.

DSN “Delivery Status Notification”. A message generated automatically to notify senders of prob-lems or failure to deliver an e-mail.

Daemon A long-running UNIX program that typically starts at system boot and continues running inthe background until the system is shut down. Roughly corresponds to a “service” on Windows.

Envelope Mail messages often have headers specifying the sender (the “From:” header) and recipi-ents (typically the “To:” header.) However, SMTP has a completely separate set of commandsfor specifying the sender and recipients. The sender and recipients specified in the SMTP com-mands are referred to as the envelope sender and envelope recipients, and do not necessarilymatch the information in the message headers. CanIt-Domain-PRO uses both the Header Fromand Envelope Sender address in Sender and Domain rules. It always uses only Envelope Recip-ients in its recipient rules.

Envelope Sender The sender address used in the “MAIL FROM” SMTP command. This is notnecessarily the same as the Header From address. Most email readers display the Header Fromaddress rather than the Envelope Sender address.


http://www.dkim.org/

https://dmarc.org/

1.4. DEFINITIONS 23

Hash An algorithm that computes a short “signature” given a chunk of data. Different inputs arevery likely to yield different signatures, so that a signature can be considered as a short-handidentifier for the original data.

Header From The sender address used in the “From:” header of an email message. This is the senderaddress displayed by most mail readers. See Envelope Sender for information about the SMTPsender address.

Greylisting A technique to block spam from certain spam-sending software. It works by issuing aTemporary Failure Code the first time an e-mail arrives from an unknown sender and IP address.Legitimate SMTP servers will retry, allowing the message to be delivered. Some spam-sendingsoftware does not retry, and messages sent by such software will be blocked without any content-scanning if greylisting is enabled.

Joe-Job A technique in which spammers fake the sending address to be that of an innocent victim,who often receives DSNs (see “DSN”) and complaints.

Malware is software designed with a malicious purpose in mind. Examples of malware are viruses,trojans, and keyloggers.

MIMEDefang is a free (GPL’d) e-mail scanning program that integrates with Sendmail’s Milter API.It forms the basis for CanIt.

MIME “Multipurpose Internet Mail Extensions”. A set of rules for encoding different types of at-tachments as plain-text messages for transmission over SMTP.

Milter is a Sendmail interface that allows external programs to listen in on the SMTP dialog, andpotentially modify Sendmail’s actions and SMTP responses.

Permanent Failure Code Also called reject, this is a code sent to a relay host telling it that e-mailtransmission has failed and will not succeed. (For example, this code is sent if someone tries tosend e-mail to a nonexistent user.) The relay host typically e-mails a failure notification to theoriginal sender and discards the message.

Phishing An attack in which someone forges e-mail pretending to be from a security organization,a bank, etc. and convinces naive users to reveal sensitive information like user-names andpasswords.

PostgreSQL A free and open-source SQL database heavily used by CanIt-Domain-PRO.

Ransomware is a specific type of malware. It typically makes changes on your computer that arealmost impossible to undo (such as encrypting all your files) and then demands payment withina short period of time to undo the damage.

Ratware is software dedicated to sending out large volumes of spam.

RBL “Real-time Blocklist”. A DNS-based system for checking in real-time whether or not hosts ordomains should be blocked. Sometimes referred to as “DNS Blocklist” or DNSBL.

RPTN is the Roaring Penguin Traning Network. This is a system whereby multiple CanIt-Domain-PRO installations can share Bayes training data.



RSS stands for “Really Simple Syndication” and is a format for publishing “news feeds” on the Web.CanIt-Domain-PRO can produce an RSS feed showing pending incidents.

Realm Administrator is a user with administrative privileges in a realm. Unlike the System Admin-istrator, a Realm Administrator can only administer his or her own realm.

Realm is a “virtual CanIt-PRO”. Within a realm, realm administrators can create streams for end-users, and streams in one realm are independent of streams in another realm.

Relay Host When a mail server wishes to transmit e-mail to your server using SMTP, it establishesa connection with your mail server. The machine attempting to transmit mail to your server iscalled a relay host.

REST Representational State Transfer. An architectural style for interacting with an API over HTTPor HTTPS. CanIt-Domain-PRO’s API is REST-based.

Root Privileges A CanIt-Domain-PRO user with root privileges can create other users and configurebasic operating parameters. Also, he or she can edit other users’ preferences and stream settings.

SMTP Dialog During the course of e-mail transmission, the two ends of an SMTP connection trans-mit commands and results back and forth. This conversation is called the SMTP dialog.

SMTP “Simple Mail Transfer Protocol”, as described in Internet RFC 2821. This is the protocol usedto transmit e-mail over the Internet.

SPF stands for “Sender Policy Framework”. It is a mechanism that allows a domain’s administratorto list which hosts are allowed to originate e-mail claiming to come from that domain. For moredetails, please see http://www.openspf.org.

SRS stands for “Sender Rewriting Scheme”. It is used in conjunction with SPF to avoid spurious SPFfailures when a CanIt-Domain-PRO machine forwards mail to a back-end server that performsSPF checks. For a description of SRS, please see http://en.wikipedia.org/wiki/Sender_Rewriting_Scheme.

Sender’s Domain This is the domain part (everything after the @ sign) in the sender’s e-mail address.

Sendmail A UNIX-based program for sending and receiving e-mail. Sendmail is designed to routemail from one mail server to another.

Spam Score A numerical score computed by CanIt-Domain-PRO that rates the likelihood that a mes-sage is spam.

Stream is a “virtual CanIt” machine offered by CanIt-PRO. If an incoming e-mail arrives for morethan one recipient, and the recipients each wish to have his or her own private spam quarantine,CanIt-PRO re-mails the original message so each recipient has his or her own copy, and candispatch it as he or she sees fit.

Syslog A UNIX program that centralizes the logging of messages from various system daemons.

System Administrator is a user with administrative privileges in the base realm. The System Ad-ministrator is responsible for overall administration of the CanIt-Domain-PRO installation.


http://www.openspf.org

http://en.wikipedia.org/wiki/Sender_Rewriting_Scheme


1.4. DEFINITIONS 25

Tempfail See “Temporary Failure Code”

Temporary Failure Code Also called tempfail, this is a code sent to a relay host telling it that e-mailtransmission has failed temporarily, and it should retry in a little while. Typically, the relay hostretains the e-mail message in a spool directory and retries transmission periodically. The hosteventually gives up after a certain period (typically, a few days) has elapsed without successfultransmission.

Ticker A CanIt-Domain-PRO program that runs periodic maintenance tasks.

Ticker Host In a CanIt-Domain-PRO cluster consisting of more than one machine, exactly one hostis designated to run the Ticker tasks. That host is called the Ticker Host.


Chapter 2

Operation

2.1 Principles of Operation

CanIt-Domain-PRO watches each incoming SMTP message and operates as follows. Because differ-ent recipients can have different settings, CanIt-Domain-PRO makes the following decisions at RCPTtime (once the recipient is known):

• If the SMTP connection is from a blocked host, the RCPT command is rejected.

• If the message sender is blocked (or the domain is blocked), the RCPT command is rejected.

• Otherwise, the message is collected and scanned.

After CanIt-Domain-PRO has scanned the message, it performs the following operations:

• Messages containing dangerous files (such as viruses) are discarded or rejected, depending onwhich option you choose.

• If the sender, relay host or domain are always-allowed, the message is accepted without beingscanned for spam.

• Many spam-detection rules are applied to the message. If the message is judged not to be spam,it is accepted and the SMTP transaction succeeds. Otherwise, CanIt-Domain-PRO will hold themessage locally.

For messages judged to be spam, CanIt-Domain-PRO takes the following steps:

• A unique ID is calculated by running the message body through a special hash function. Thehash calculation is designed to be resistant to some forms of trivial message modification.

• The ID is looked up in a database.

1. If the ID is not found in the database, it is entered as a pending message. CanIt-Domain-PRO will either hold a copy of the message locally or send a temporary failure code to theSMTP sender, depending on how CanIt-Domain-PRO has been configured.


28 CHAPTER 2. OPERATION

2. If the ID is in the database with status pending, CanIt-Domain-PRO may either save alocal copy or return a temporary failure code to the SMTP sender, depending on howCanIt-Domain-PRO has been configured.

3. If the ID is in the database with status spam, a permanent rejection code is sent to theSMTP sender.

4. If the ID is in the database with status not-spam, the message is accepted for delivery.

The flow of mail through CanIt-Domain-PRO is summarized in Figure 2.1. Note that this is theconceptual flow; in reality, several optimizations are performed that would only complicate the figure.See also Figures 2.2 on page 29 and 2.3 on page 30 for more accurate details about block and allowrules.

RCPT Command

Block rule?

Accept

RCPT

Reject

RCPT

Proceed

to DATA

End of DATA

Virus?

Allow−always?

Message

Discard

Message

Deliver

Looks

Like Spam?

Message

Deliver

Message

Hold

Y

N

Y

N

N

Y

N

Y

Figure 2.1: Flow of Mail through CanIt-Domain-PRO

2.2 Interaction between Allow Rules and Block Rules

CanIt-Domain-PRO must prioritize allow and block rules. For example, suppose a sender is alwaysallowed, but the host the message comes from is blocked. What should CanIt-Domain-PRO do?

2.2.1 RCPT TO: Actions

At the SMTP RCPT TO: command, CanIt-Domain-PRO examines the envelope sender and SMTPrelay address, and makes decisions according to Figure 2.2.


2.2. INTERACTION BETWEEN ALLOW RULES AND BLOCK RULES 29

REJECT

Sender

Always−allowed?

Always−allowed?

Domain

Domain

Blocked?

REJECT

REJECT

ALLOW

ALLOW

REJECT

Always−allowed?

Relay

REJECT

ALLOW

ALLOW

Relay

Blocked?

Y

N

Y

N

Relay on

Reject RBL?

Y

N

Sender

Blocked?

Y

N

Y

N

Y

N

Y

N

Recipient?Y

N

Start

Invalid

Figure 2.2: RCPT TO: Decision

Here are the steps illustrated in Figure 2.2. They determine the response to the RCPT TO: command.The first rule that matches returns the result; subsequent rules are not tested.

1. If the recipient is blocked, the command is rejected. Blocked recipients can never receive e-mail.

2. If the recipient has opted out of spam-scanning, the command is accepted.

3. If the sender address is blocked, reject the command with an SMTP failure code.



4. If the sender address is always allowed, accept the command. (That is, permit the SMTP trans-action to continue. The message may be rejected later for other reasons.)

5. If the domain of the sender is blocked, reject the command.

6. If the domain of the sender is always allowed, accept the command.

7. If the sending relay’s IP address is blocked, reject the command.

8. If the sending relay’s IP address is always allowed, accept the command.

9. If the sending relay is on a real-time blocklist for rejection, then reject the command.

10. Otherwise, accept the command.

2.2.2 Post-DATA Actions

After the SMTP “DATA” command has transmitted the entire message, CanIt-Domain-PRO hasenough information to determine a spam score. At this point, it makes decisions according to Fig-ure 2.3.

START

Virus Handling

Accept Message

Reject Message

Hold in Trap

Accept Message

Reject Message

Hold in Trap

Bad Attachment

Handling

Reject Message

Accept Message

Hold in Trap

Accept Message

Hold in Trap

Hold, Tag

or Reject

Virus Found?

SenderAlways allowed?

Bad MIME typeor Extension?

Sender

Sender"Hold"?

Blocked?

Y

Y

Y

Y

N

N

N

N

"Hold"RBL Rule?

High SpamScore?

Y

Y

N

N

Always allowed?

Blocked?

"Hold"?Relay

Relay

Relay

Always allowed?Domain

Domain

Domain

Blocked?

"Hold"?

Y

N

Y

N

Y

Y

Y

Y

Y

N

N

N

N

N

Figure 2.3: Post-Data Decision


2.2. INTERACTION BETWEEN ALLOW RULES AND BLOCK RULES 31

Here are the steps illustrated in Figure 2.3. They determine the response to the DATA command. Thefirst rule which matches returns the result; subsequent rules are not tested. (There is one exception:If a “Hold Sender”, “Hold Domain” or “Hold Relay” rule is hit, but the message scores over theauto-reject threshold, the message is rejected rather than held for review.)

When a message is “held in the quarantine”, the message will be held by CanIt-Domain-PRO forreview. To the sending SMTP relay, it appears as if the message was delivered successfully.

When a message is “rejected”, the sending relay receives an SMTP failure code. If the message beingrejected was held within CanIt-Domain-PRO, it is simply discarded.

When a message is “accepted”, it is simply delivered as usual.

1. If a virus was found in the message, then the action depends on the virus-handling setting.Here’s what happens for the various settings:

• Hold/Tag – the message is held in the quarantine (or tagged in a tag-only stream.)

• Reject – the message is rejected with an SMTP failure code.

• Discard – the message is discarded. An SMTP success code is returned.

• Accept – processing continues to step (2) below.

2. If a bad MIME part or filename extension was found, then if the bad part has a “Reject” setting,the message is rejected. Otherwise, the message is held in the quarantine (or tagged in a tag-onlystream.)

3. If the user has opted-out of spam-scanning, the message is accepted

4. If the sender is always allowed, the message is accepted.

5. If the sender is blocked, the message is rejected. It may seem superfluous to check for a blockhere, given that the block was checked during the RCPT command. However, by the DATAcommand, we have the From: header, and CanIt-Domain-PRO applies sender checks to theFrom: header address also.

6. If the sender has a “Hold/Tag” setting, the message is held in the quarantine (or tagged in atag-only stream.) However, if it scores over the auto-reject threshold, it will be rejected.

7. If the domain is always allowed, the message is accepted.

8. If the domain is blocked, the message is rejected. Again, at this point, CanIt-Domain-PRO canmake use of the From: header address.

9. If the domain has a “Hold/Tag” setting, the message is held in the quarantine or tagged. How-ever, if it scores over the auto-reject threshold, it will be rejected.

10. If the relay is always allowed, the message is accepted.

11. If the relay has a “Hold/Tag” setting, the message is held in the quarantine or tagged. However,if it scores over the auto-reject threshold, it will be rejected.



12. If the relay is on a “Hold/Tag” real-time DNS blocklist, the message is held in the quarantine ortagged.

13. If CanIt-Domain-PRO is in “Tag Only” mode, the message is tagged (if it looks like spam) andaccepted.

14. If the spam score is equal to or above the auto-reject threshold, the message is rejected. Oth-erwise, if the spam score is equal to or above the spam threshold, the message is held in thequarantine.

15. Otherwise, the message is accepted.

2.3 Streaming

Because CanIt-Domain-PRO allows different recipients to have different spam-processing rules, anincoming message for more than one recipient must be streamed.

The diagram in Figure 2.1 shows what happens to messages after they have been streamed. If anincoming message arrives for more than one stream, copies are re-mailed to recipients in each stream,and the original message is discarded. Then, each re-mailed message follows the flow in Figure 2.1,with some minor differences that will be explained later.

In Figure 2.1, all of the block and allow decisions are unique to a stream. It is perfectly feasible forone stream to always allow a sender, a second stream to block it, and a third stream to do neither.

Messages that are streamed and re-mailed are not held by issuing a temporary-failure code, becausethey would then reside in your own mail spool and waste resources during repeated sending attempts(until they are approved or rejected.) Instead, held messages are stored in the database, and re-mailedif approved or discarded if rejected.

2.4 How Addresses are Streamed

CanIt-Domain-PRO can map e-mail addresses to streams using the following techniques:

Database CanIt-Domain-PRO maintains a table of address-to-stream mappings in the Address Map-ping Table. If you choose the Database technique, then this table is consulted to performthe mapping. You hand-enter the mappings between addresses and streams. In addition, theDatabase technique allows a “wildcard” lookup if the original lookup does not exist.

AsIs This method simply uses the entire e-mail address as the stream name, after stripping angle-brackets and converting to lower-case. Therefore, [email protected] gets mapped [email protected],

ChopDomain This method simply chops the domain part off the e-mail address. Therefore,[email protected] gets mapped to xzyyz.

ChopUser This method chops the user part off the e-mail address. Therefore,[email protected] gets mapped to example.com.


2.5. HOW STREAMING METHODS ARE CHOSEN 33

Program This method runs the account-info program to determine the stream. Please see Sec-tion 7.2.5 on page 157 for details.

User Lookup You can create so-called “User Lookups” that permit you to use LDAP or arbitraryscripts to map addresses to streams. These are described in Section 7.2.

Note: No matter what stream method you choose, an exact-match database lookup is always done first. Thislets you override the mapping for special cases. For example, if you host only a single domain, thenthe ChopDomain method is probably fine for most addresses. However, if you also host mailinglists, you’d like to stream spam for the lists to the mailing list owners. In that case, you can addspecial mappings mapping [email protected] to joe-owner, (where joe-owner isthe person responsible for list-name.)

Because the Program method is somewhat inefficient, CanIt-Domain-PRO caches results in thedatabase table. This improves efficiency while retaining flexibility. By default, cached entries arevalid for 24 hours, but you can adjust the timeout.

2.5 How Streaming Methods are Chosen

Each domain can be streamed using its own method. To select a streaming method, CanIt-Domain-PRO first looks up the domain in the Domain Mapping Table. This table holds a list of streamingmethods for each domain. If the lookup fails, CanIt-Domain-PRO looks up the wildcard entry “*” inthe Domain Mapping Table and uses that method to stream the address.

Figure 2.4 illustrates how addresses are streamed.



Incoming Mail for

[email protected]

method = lookup

"example.com" in

Domain Mapping Table

method = lookup

"*" in

Domain Mapping Table

method found?

method found?

stream = lookup

in Address Mapping Table

"[email protected]" (followed

by "user@*" if not found)

stream found?

method =ChopDomainChopUseror AsIs?

Return stream

stream = adjust address

method = "Database"

method =

Program? to determine

local user

Run account−info script Cache stream in

Address Mapping

Table

method =

LDAP in LDAP directory.

Look up stream

stream = lookup

Address Mapping Table

"*" in

stream found?

stream = lookup

Address Mapping Table

stream found?

stream = "default"

Y

N

Y

N

"*@example.com" in

Y

N

N

Y N

Y

N

Y

N

Y

Y

N

Figure 2.4: Address Streaming


2.6. STATUS OF MESSAGES 35

Figure 2.4 looks complicated, but the streaming process is very flexible, and actually quite simple.Here is a description of the figure, with some more details that would crowd the figure too much.

1. For an incoming message to [email protected], CanIt-Domain-PRO first looks up exam-ple.com in the Domain Mapping Table. If that lookup succeeds, CanIt-Domain-PRO will havea method (ChopDomain, ChopUser, Program, Database or a user-lookup name), and CanIt-Domain-PRO proceeds to Step 4.

2. If the lookup fails, the leading component of the domain name is dropped (ie: “subdo-main.example.com” becomes “example.com”) and we retry Step 1 with the shorter name.

3. If lookups on all domain components fail, CanIt-Domain-PRO looks up * in the Domain Map-ping Table. This allows you to set a default streaming method for all domains. If that lookupfails, the method defaults to Database.

4. Regardless of the method chosen, CanIt-Domain-PRO looks up [email protected] in the Ad-dress Mapping Table. If an exact match is found (and it is not expired if it is a cached entry),the result of that lookup is used as the stream. If the exact match is not found, but a wildcarduser@* is found in the Address Mapping Table, the result of that lookup is used as the stream.

5. Otherwise, CanIt-Domain-PRO determines the stream as follows:

• If the method is ChopDomain, the @example.com part is deleted, and the stream becomesuser.

• If the method is ChopUser, the user@ part is deleted, and the stream becomes exam-ple.com.

• If the method is AsIs, the entire e-mail address [email protected] is used as the streamname.

• If the method is Program, CanIt-Domain-PRO runs the account-info program asdescribed in Section 7.2.5.

• If the method refers to a user-lookup, then the user-lookup is invoked to determine thestream. See Section 7.2 for details.

If the stream determination succeeded (AsIs, ChopDomain and ChopUser always succeed;Program fails if the program produces no output), then the stream is returned. Additionally, thestream may be cached in the Address Mapping Table.

6. If the previous step failed to determine a mapping method, or the method was set to Database,CanIt-Domain-PRO looks up *@example.com in the address mapping table. This allows you tomap all addresses in a particular domain to a stream. If that fails, as a last resort, CanIt-Domain-PRO looks up * in the address mapping table. If that final lookup fails, then a special streamnamed default is used.

2.6 Status of Messages

Every message in the database has one of three statuses. The status names and their meanings are:



pending Messages enter pending state when they arrive, and remain there until they are marked asspam or nonspam. These messages are displayed in the Web-based “Pending Messages” list.

spam The spam-control officer can mark a message as spam. If a message marked as spam is re-ceived, a rejection notice is sent to the sending mail server, and the message is not delivered.

not-spam The spam-control officer can mark a message as not-spam. If a message marked as not-spam is received, it is delivered as usual.

2.6.1 Secondary MX Relays

Many organizations have secondary MX hosts that queue mail if the primary host is down. They thenrelay the queued mail when the primary MX host comes back up. Ideally, CanIt-Domain-PRO shouldrun on all of your MX hosts. However, if it can only run on your primary MX host, then all other MXhosts should relay to the CanIt-Domain-PRO machine. You should then tell CanIt-Domain-PRO the IPaddresses of the secondary MX hosts via the “Known Networks” facility so that CanIt-Domain-PROcan use the Never Tempfail handling for messages from those hosts. (There is no point in keepingmail queued and retransmitted on your secondary MX hosts; it’s better to accept and hold the messageon the CanIt-Domain-PRO machine.)

2.7 The Database

The incident database is key to the correct operation of CanIt-Domain-PRO. Three different agentsoperate on the database as shown in Figure 2.5:

Incidents

Database

Web−Based

GUIPeriodic Jobs

CanIt Filter

Figure 2.5: Database Agents

The agents operating on the database are:

• The CanIt-Domain-PRO Filter – This is the portion of CanIt-Domain-PRO that integrates withSendmail and disposes of spam messages.


2.8. REMAILING MESSAGES 37

• The Web-Based GUI – This is used by users or administrators to mark messages as spam orlegitimate. The Web-Based GUI also lets you monitor the levels of spam and take action againstspecific senders, domains or relay hosts.

• Periodic Jobs – These housekeeping jobs perform operations like moving expired pending mes-sages into spam status and purging very old messages from the database. Periodic jobs may bestarted from one of two places:

1. The /usr/share/canit/scripts/canit.cron script, which should be run oncea night.

2. As part of the operation of the CanIt-Domain-PRO daemon (canitd). Canitd is a daemonthat starts on bootup and runs continuously, performing background maintenance tasks.

2.8 Remailing Messages

On occasion, CanIt-Domain-PRO will be forced to remail a message after discarding the original. Thefollowing scenarios cause remailing:

1. If a message comes in for recipients in more than one stream, CanIt-Domain-PRO generates onenew copy for each stream and mails out the copies. The original message is then discarded. Youmay see a message in the log file indicating that the message has been discarded; don’t panic.The copies are safely queued.

2. If a Pending message is held in the database and subsequently approved for release, CanIt-Domain-PRO fetches the message body from the database and remails it. This always takesplace on the designated ticker host, no matter which host processed the original message.

In all cases when CanIt-Domain-PRO remails a message, the message goes into Sendmail’ssubmission queue (most likely in the queue directory /var/spool/clientmqueue or/var/spool/mqueue-client. The message is only processed on the next run of the submis-sion queue. For this reason, you should keep the submission queue interval short (on the order of aminute or two.) On CanIt-Domain-PRO appliances, the submission interval is automatically config-ured for you. On other platforms, consult your system’s documentation for details on how to shortenSendmail’s submission queue interval.


Chapter 3

Realms

3.1 Introduction to Realms

CanIt-Domain-PRO has three levels of administrative control:

1. The System Administrator administers all aspects of CanIt-Domain-PRO and is responsible forsetting up and provisioning the system.

2. A Realm Administrator administers settings and rules for a given realm. A realm encompassesone or more Internet domains. The realm administrator is responsible for provisioning streamswithin his or her realm. A realm administrator is said to have root privileges within a realm.

3. A Stream Owner administers settings and rules for his or her own stream. A stream owneris typically an end-user or a person responsible for administering a small group of e-mail ad-dresses.

The administrative levels are illustrated in Figure 3.1 below:

Realm Administrator Realm Administrator Realm Administrator

Stream Owner Stream Owner

Stream Owner Stream Owner

System Administrator

Realm 2Realm 1 Realm N

Stream 1 Stream N

Stream 1 Stream N

Figure 3.1: Administrative Levels


40 CHAPTER 3. REALMS

3.2 Realm Names

A realm name can consists only of letters, numbers, dashes and underscores. That is, only the follow-ing characters can appear in a realm name:

A B C D E F G H I J K L M N O P Q R S T U V W X Y Za b c d e f g h i j k l m n o p q r s t u v w x y z0 1 2 3 4 5 6 7 8 9 0 -

Realm names are case-sensitive; a realm named REALM-ONE is different from realm-one.

3.2.1 The base Realm

The realm named base is special. This realm always exists and cannot be deleted. Any user withroot privileges in the base realm is considered an overall CanIt-Domain-PRO system administrator,and can access any realm and setting.

In other words, a realm administrator of the base realm is an overall CanIt-Domain-PRO administra-tor.

3.3 Creating Realms

Note: This section describes features that only the CanIt-Domain-PRO System Administrator can use.

Click on Setup and then Realms. The Realm Screen appears:

Figure 3.2: Realm Screen

To create a realm:

1. Enter the realm name in the Realm box.

2. Enter a short description in the Description box.


3.4. REALM MAPPINGS 41

3. If you wish to enter an expiry date, do so in the Expiry box. See Section 3.6 for details aboutrealm expiry.

4. Normally, all realms you create have the base realm as a parent realm. If you wish to set arealm’s parent to something else, select a realm name from the Parent pull-down menu. SeeSection 3.7 for details about realm hierarchy.

5. Click Submit Changes.

To delete a realm:

1. Enable the Delete? checkbox for the realm you wish to delete.


Note that it is not possible to delete the base realm.

3.4 Realm Mappings

Note: Only the CanIt-Domain-PRO System Administrator can create new realm mappings. Realm adminis-trators can delete realm mappings (irrevocably) or remap a domain from one realm to another.

To associate a domain with a realm, CanIt-Domain-PRO uses a Realm Mapping Table. To access thistable, click on Setup and then Realm Mappings. The Realm Mappings screen appears:

Figure 3.3: Realm Mappings

In this example, the domains roaringpenguin.com and roaringpenguin.ca are bothmapped to the roaringpenguin realm, while artandframingsolutions.com is mappedto afs. If CanIt-Domain-PRO accepts mail for other domains, then they will be mapped to the base



realm. Any domain without an explicit realm mapping will be mapped to base. (The rules for realmmapping are summarized in Section 3.5.)

To add a realm mapping:

1. Enter the domain name in the Domain box.

2. Select the realm name in the Realm box. Note that you must create realms before you can addmappings to them.


To delete a realm mapping:

• Enable the checkbox next to the mapping you wish to delete.

• Click Submit Changes.

3.5 Determining the Realm

CanIt-Domain-PRO determines the realm for e-mail addresses and user names as follows:

3.5.1 Mapping an Address to a Realm

1. Given an e-mail address of the form [email protected], CanIt-Domain-PRO looks up thedomain (domain.com) in the Realm Mapping Table and uses the realm found in the table.

2. If no realm was found in Step 1, the address is placed in the base realm.

Note: The addresses postmaster, postmaster@localhost and postmaster@machine nameare always mapped to the base realm, no matter what. (Here, machine name is the name of thehost processing the email.)

3.5.2 Mapping a Login Name to a Realm

1. If a user’s login name is of the form [email protected], then CanIt-Domain-PRO uses theprocedure described in Section 3.5.1 to determine the realm.

2. If a user logs in with a name of the form realm:user, then CanIt-Domain-PRO uses realmas the realm name.

3. Otherwise, CanIt-Domain-PRO uses the default realm as configured in thesite/config.php configuration file. If no default realm is set in that file, thenCanIt-Domain-PRO uses base as the realm name.


3.6. REALM EXPIRY 43

3.6 Realm Expiry

When you create a realm, you can set an expiry date. Whenever the realm administrator logs in toCanIt-Domain-PRO, he or she will receive a warning starting 30 days prior to the expiry date. If youare hosting CanIt-Domain-PRO realms on behalf of third-parties, this is a good way to remind them torenew their subscription. The expiry date normally has no other effect (in particular, CanIt-Domain-PRO will continue filtering mail as usual after the expiry date) and is intended only as a renewalreminder. If you do not set an expiry date, then the realm never expires.

3.6.1 Suspending Service to a Realm

While the expiry date field normally has no effect, if you set the expiry to the “magic” date1990-01-01, then all service to the realm is suspended. What this means is:

• No users in that realm will be able to log in.

• All mail to anyone in the realm will be permanently rejected with a “Service suspended” errormessage.

Suspending service to a realm is a drastic step since it causes all mail to bounce. Please use it only asa last resort.

3.7 Realm Hierarchy

Realms normally have the base realm as their parent. However, if you are reselling CanIt-Domain-PRO services to others who wish to have their own set of realms for their customers, you can createa realm hierarchy. A realm administrator has access to his or her own realm and all realms under it.Consider Figure 3.4:



base

cust−1 cust−2

subcust−2−1 subcust−2−2

subcust−2−1−1

Figure 3.4: Realm Hierarchy Example

In the example in Figure 3.4, the parent of cust-1 and cust-2 is base. The parentof subcust-2-1 and subcust-2-2 is cust-2, and the parent of subcust-2-1-1 issubcust-2-1.

• The administrative user in the base realm can access all realms.

• The administrator in cust-1 can only access the cust-1 realm.

• The administrator in cust-2 can access subcust-2-1, subcust-2-2 andsubcust-2-1-1.

• The administrator in subcust-2-1 can access subcust-2-1 and subcust-2-1-1.

• The administrator in subcust-2-2 can only access subcust-2-2.

• The administrator in subcust-2-1-1 can only access subcust-2-1-1.

In the Realms screen (Figure 3.2), click on Tree View to see a hierarchical view of the realms. Youcan restrict the view to a subtree of the entire hierarchy by selecting the root of the tree from the Treeroot pull-down menu.

3.8 Realm Custom Fields

CanIt-Domain-PRO allows you to create up to four custom fields so you can associate various piecesof information with a realm. For example, you may wish to include a customer number with eachrealm. To configure custom fields, click on Setup and then Realms. In the realm display, click onCustom Fields. The Custom Fields screen appears:


3.8. REALM CUSTOM FIELDS 45

Figure 3.5: Realm Custom Fields

To create custom fields:

1. Enter the name of the field in the Name box.

2. If you wish to have the field displayed specially, enter a format string in the Format box. Thisstring must contain exactly one %s sequence; this will be replaced by the value of the customfield. In the example in Figure 3.5, Custom Field 2 (AccountID) will be displayed as a hyperlink,presumably to an accounting system.

3. Click Submit Changes to make the changes take effect.

Any custom fields you create are displayed as additional columns in the Realms screen (for the CanIt-Domain-PRO administrator only!). To remove a custom field, simply make the Name column blank.


Chapter 4

Streams

4.1 Introduction to Streams

The stream is a central concept in CanIt-Domain-PRO. Understanding streams is essential to un-derstanding CanIt-Domain-PRO. Please be sure to read this chapter before configuring a productionCanIt-Domain-PRO server.

4.2 Realms

A realm is a collection of Internet domains, all of whose anti-spam settings and quarantines are provi-sioned by a Realm Administrator. Within a realm, there may be many streams. Two streams with thesame name can coexist in different realms; CanIt-Domain-PRO will consider them to be two differentstreams.

4.3 The Definition of a Stream

A stream is a collection of rules and policies. Each stream in CanIt-Domain-PRO can have its ownrules, settings, thresholds and policies.

Associated with each stream is a quarantine. A quarantine consists of messages that have been heldbased on the streams settings. For example, a message can be held because of its spam score, orbecause it contains a suspicious MIME type.

4.4 Users and E-Mail Addresses

Under many circumstances, a single e-mail address corresponds to a single user. For example, thee-mail address [email protected] corresponds to the single user dfs.

However, most mail setups are more complicated than this. The first complication comes fromaliases. For example, the user dfs may have, in addition to his normal e-mail address, aliases


48 CHAPTER 4. STREAMS

like [email protected] and [email protected]. We would mostlikely want the same settings and policies to apply to all three aliases.

Another complication comes from list addresses. For example, the e-mail [email protected] does not correspond to any particular user. Instead, it is a listalias that expands to several users. It might make sense to have a separate set of policies for salesthan for real users, or it might make sense to assign the policies used by one of the recipients on thesales list.

As we see above, the mapping between users and e-mail addresses is not simple. A single e-mailaddress may result in delivery to several users (the sales example), or a single user may have severale-mail addresses that all deliver to the same place (the aliases example.)

Streams were created to give you the flexibility of assigning policies. They act as an intermediatecontainer between e-mail addresses and actual users, and let you assign policies any way you choose.As an example, consider Figure 4.1:


4.4. USERS AND E-MAIL ADDRESSES 49

[email protected]

[email protected]

[email protected]

dfs

paul

StreamE−Mail Address User−ID

dfs

paul

(a)

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

dfs

StreamE−Mail Address User−ID

dfs

[email protected]

[email protected]

(b)

paul paul

sales

Figure 4.1: Streaming Scenarios

Note that streaming affects only how CanIt-Domain-PRO directs mail for rule and quarantine pur-poses. Streaming does not alter the ultimate delivery address; normally, CanIt-Domain-PRO delivers



mail to the back-end server without altering recipient addresses at all.

We assume that there are two users, dfs and paul. We assume that dfs has the three aliases shown,and that the sales address actually gets delivered to both dfs and paul.

In Figure 4.1(a), all mail for dfs’s aliases go into the dfs stream. Mail for paul goes into the paulstream. Furthermore, mail for sales also goes into paul. Although mail for sales is delivered totwo users, all of the settings and policies are controlled by the paul stream, and paul is responsiblefor clearing the quarantine.

In Figure 4.1(b), sales has its own stream. It can thus have different settings and rules from eitherpaul or dfs. Furthermore, both paul and dfs are given access to the stream, so either of thoseusers can adjust the settings and check the quarantine for sales.

4.5 Mapping

When e-mail comes in, each recipient address is mapped to a stream. We call this process addressmapping. Once the stream is determined, CanIt-Domain-PRO knows which settings and rules to applyfor that recipient. The process by which CanIt-Domain-PRO maps addresses to streams is illustratedin Figure 2.4 on page 34.

An e-mail address is mapped to a stream in a three-step process:

1. The domain part of the address (everything after the “@” sign) is looked up in the Realm Map-ping Table. This lookup determines the realm to which the address belongs.

2. The domain part of the address is looked up in the Domain Mapping Table. This lookupresults in a method by which to map the address to a stream. Note that CanIt-Domain-PRO looks up the mapping method using a combination of the realm (determined in step1) and the domain. The combination of realm and stream determined in this step is writtenrealm name:stream name

3. Once the method has been determined, then the address is mapped to a stream using the appro-priate method. Details are in Section 5.14 on page 81.

Note: If there is an exact match for an email address in the Address Mapping Table (under Setup :Address-to-Stream Mappings) then it is always used, overriding any mapping method. Fur-thermore, if there is no exact match, but there is an entry for user@*, then that entry too isused, overriding any mapping method.

4.6 The Home Stream

When a user logs in to the Web interface, CanIt-Domain-PRO must associate a stream with the username. By default, CanIt-Domain-PRO chooses a stream with the same name as the user’s login—thisis called the home stream. For example, the user dfs would automatically be sent to the stream dfsupon login. However, it is possible to give users access to additional streams, and to change the default


4.7. THE “DEFAULT” STREAM 51

login stream. Also, it is possible to change the user’s home stream with the account-info script(Section 7.2.5).

Note: Stream names are case-sensitive. Thus, a stream called dfs is completely separate from a streamcalled DFS.

4.7 The “default” Stream

CanIt-Domain-PRO treats the stream named default specially in several ways:

• When the database initialization script runs, it sets the login stream for the CanIt-Domain-PROadministrator to default.

• If a stream mapping cannot be found for an address, the address is mapped to default.

• Any blocks, allow rules and other rules defined in the default stream are inherited by all otherstreams. (However, stream owners can turn this inheritance off if they wish.) Note that inCanIt-Domain-PRO, rules for a stream example-stream in realm example-realm aresearched up through the realm hierarchy.

1. Search for the rule in example-realm:example-stream.

2. If not found, search in example-realm:default.

3. If not found, search in example-realm’s parent realm in the default stream. Con-tinue looking up the realm hierarchy until base:default is reached.


Chapter 5

CanIt-Domain-PRO Setup

5.1 Accessing The Web Interface

Using your Web browser, open the URL where you installed the CanIt-Domain-PRO web pages.

For example, if your server is mailserver.mydomain.com and you installed the GUI in thedirectory canit under your Apache document root, the URL to open would be:

http://mailserver.mydomain.com/canit/

(By default, our binary packages and our Debian-based appliances put the web pages athttp://machine.yourdomain.net/canit/)

5.1.1 License Key Screen

The very first time you log in, you will see the License Key Screen (Figure 5.1):

Figure 5.1: License Key Screen

Enter or cut-and-paste your license key into the entry box and click Submit Key. The license keyincludes all the text starting from License and continuing to the end of the string of letters andnumbers after Check=.


54 CHAPTER 5. CANIT-DOMAIN-PRO SETUP

5.1.2 Login Screen

Once the license key has been entered, navigating to the CanIt-Domain-PRO URL reveals the LoginScreen (Figure 5.2):

Figure 5.2: Login Screen

Log in using the name and password you selected when you initialized the CanIt-Domain-PROdatabase. (See Section J.1 on page 371 if you’ve forgotten the password.)

In the Installation Guide example, we used “admin” and “secret”. If you have a CanIt-Domain-PROappliance, the defaults are “admin” and “canit”. (Naturally, you should change the password beforeconnecting your CanIt-Domain-PRO appliance to the Internet!)

Normally, CanIt-Domain-PRO will set a session cookie in your browser. This means that if you closeyour browser, your session will automatically end. If you want CanIt-Domain-PRO to remember yoursession even if you close the browser, enable the “Remember Me” checkbox. This puts a cookie thatlasts longer (by default, 7 days) on your computer. Do not use the “Remember Me” option on a publiccomputer; you should only use it on a workstation to which you alone have access.

Once logged in, you should see the CanIt-Domain-PRO welcome screen:


5.2. THE SETUP MENU 55

Figure 5.3: Welcome Screen

5.2 The Setup Menu

The Setup main menu entry contains sub-entries for various parts of basic CanIt-Domain-PRO setup.Under the Setup menu, you will find:

• Wizards – a collection of tools for easily configuring certain common scenarios.

• License Key – a page to enter your CanIt-Domain-PRO license key.

• Verification Servers – a table allowing you to check recipients against internal servers beforeCanIt-Domain-PRO will accept them.

• Known Networks – a table allowing you to change aspects of CanIt-Domain-PRO behavior formail originating from certain known networks.

• Features – a page allowing you to turn off certain CanIt-Domain-PRO functionality to improveperformance.

• System Check – a page that performs a few simple “sanity checks” on your CanIt-Domain-PROsystem.

• Templates – a page for configuring templates that control how CanIt-Domain-PRO appendsBayesian voting information to e-mail and the format of Pending Message Notifications.

• Theme Customization – a page for customizing the CanIt-Domain-PRO “look”. Can be usedto brand CanIt-Domain-PRO.

• Domain Routing – a page for configuring e-mail routing. Please note that this link is availableonly on Debian-based appliances or on RPM installations with the appliance RPMs installed.

• HTTPS – a page for configuring HTTPS. Please note that this link is available only on Debian-based appliances. (It is not available on RPM or source installations.)



• Cluster Management – a page for viewing and managing cluster members.

• Domain Mappings and Address Mappings – two tables that tell CanIt-Domain-PRO how toconvert an e-mail address to a stream.

• Authentication Mappings and User Lookups – pages for integrating CanIt-Domain-PRO withexternal directories or authentication mechanisms. These are fully described in Chapter 7.

5.3 Wizards


The Wizards menu item allows you to ease CanIt-Domain-PRO setup by using a wizard to speedthrough choosing some basic settings. The available wizards are shown on the Setup page. Thewizards are self-documenting and guide you through the steps required to configure CanIt-Domain-PRO. However, the following wizards are important enough to warrant mention:

5.3.1 Basic Setup Wizard

The Basic Setup Wizard helps you set some basic settings essential to the operation of CanIt-Domain-PRO. On a new CanIt-Domain-PRO installation, you should follow the steps in this wizard to set somebasic settings to sensible values. It is important not to operate CanIt-Domain-PRO until you haveworked through the Basic Setup Wizard.

5.3.2 RPTN Setup Wizard

The RPTN Setup Wizard configures RPTN, the Roaring Penguin Training Network. (RPTN is amechanism for sharing Bayes data to increase scanning accuracy. See Section 8.5 on page 162 fordetails.)

5.3.3 Dictionary Attack Detection Wizard

Note: Dictionary Attack Detection works only on Linux.

A Dictionary Attack is an attack whereby an attacker tries to send mail to hundreds or thousands ofdifferent e-mail addresses within a domain in the hopes of discovering some valid addresses. CanIt-Domain-PRO (on Linux only) can react to dictionary attacks by blocking them using kernel firewallrules.

To enable dictionary-attack detection:

1. Click on Setup : Wizards and then Dictionary Attack Detection Wizard.

2. Select Yes when asked “Would you like to enable the dictionary-attack detector?” Click Next.

3. Adjust the parameters as follows:


5.4. VERIFICATION SERVERS 57

• Time span over which to track bad recipients specifies for how long CanIt-Domain-PRO will keep history. For example, if you specify 900 seconds, then CanIt-Domain-PROtracks bad recipients over the last 15 minutes.

• Number of bad recipients to trigger firewalling specifies how many bad RCPT com-mands a host must issue (within the tracking time) to be firewalled off. Continuing theexample, if you specify 5 for this parameter, then any host that issues 5 invalid RCPTcommands within 900 seconds will be firewalled off.

• Length of time in seconds to remain firewalled specifies how long a host remains fire-walled once CanIt-Domain-PRO decides it is an attacker. The default is 3600 seconds (onehour.)

4. Click Next

5. Review your settings and click Finish to make them take effect.

You may wish to exclude certain hosts from ever being banned because of bad RCPT commands. Youcan exclude such hosts by adding them to the Known Networks list (Section 5.7) with the Omit fromDictionary Attack Detection flag set.

Note: When a host is firewalled off, the Sendmail process that triggered the firewall rule will not receive anytraffic from the host. By default, Sendmail will wait one hour between commands. This is far too longif you use the dictionary-attack detector; we recommend shortening Sendmail’s Timeout.commandparameter to 5 minutes or shorter. On CanIt-Domain-PRO appliances, this configuration change hasbeen done for you. On other platforms, include the line:

define(‘confTO COMMAND’, ‘5m’)dnl

in your sendmail.mc file and rebuild sendmail.cf.

5.4 Verification Servers

If CanIt-Domain-PRO acts as a filtering server that always forwards mail on to other machines, you canhave it check recipient addresses against other machines. The internal machine that verifies recipientaddresses is called a Verification Server. The mechanism is illustrated in Figure 5.4:

1

4

2RCPT TO:<[email protected]>

Scanner Verification Server

RCPT TO:<[email protected]>

Response: 250 OK / 550 No User

3Response: 250 OK / 550 No User

Figure 5.4: Verification Server Operation

The sequence of events in Figure 5.4 is as follows:

1. An external SMTP server sends the command: RCPT TO:<[email protected]>



2. Before CanIt-Domain-PRO accepts the RCPT command, it starts an SMTP session with theVerification Server (sending a HELO and MAIL command first) and sends the same RCPTcommand to the Verification Server.

3. The Verification Server responds to the CanIt-Domain-PRO scanner with a reply code.

4. The CanIt-Domain-PRO scanner responds to the external server with the same response it re-ceived from the Verification Server.

Note: This feature only works if the internal machines fail RCPT commands for unknown users. That is, theinternal machine must be configured to reject unknown recipients during the SMTP transaction. SomeSMTP servers accept any recipient address and then later on generate a failure notification. Serversthat delay the rejection of invalid addresses in this manner will not work as Verification Servers.

Versions of Microsoft Exchange prior to Exchange 2003 will not work as verification servers.Recent Exchange versions can be configured to reject unknown recipients during the SMTPtransaction. See the instructions linked from https://www.roaringpenguin.com/recipient-verification for your version of Exchange.

In all cases, you should disable all other Exchange anti-spam features including tarpitting. (Tarpittingis a completely useless technology for a server behind a spam filter and serves only to slow downCanIt-Domain-PRO.) Make sure that the only anti-spam feature enabled on the Exchange server isrecipient filtering.

CanIt-Domain-PRO allows you to enter a list of domains and the machines that will verify mail forthe domains. (Note that this does not change your Sendmail configuration; you need to ensure thatSendmail’s mailertable routes mail appropriately.)

To edit the verification server list, click on Setup and then Verification Servers. The following pageappears:

Figure 5.5: Verification Servers

In this example, CanIt-Domain-PRO performs the following checks:

• Any recipient whose domain is blacky.roaringpenguin.com is verified against the ma-chine blacky.roaringpenguin.com

• Any recipient whose domain is canit.ca is verified against the machine mail.canit.ca


https://www.roaringpenguin.com/recipient-verification

https://www.roaringpenguin.com/recipient-verification

5.4. VERIFICATION SERVERS 59

• Any recipient whose domain is roaringpenguin.com is verified against the machinemail.roaringpenguin.com

To add a domain/server pair to the table:

• Enter the domain name in the Domain box and the server name or IP address in the Serverbox. Note that you can enter multiple verification servers in the Server box by separating thenames or addresses with commas. If you enter multiple servers, CanIt-Domain-PRO tries themin order until it receive a definite positive or negative response.

• Sometimes, your verification server may be down or unreachable. There are three approachesto deal with this situation:

– If you would like CanIt-Domain-PRO to tempfail the mail, set Action if Unavailable to“Tempfail”.

– If you would like CanIt-Domain-PRO to queue mail to addresses that have been provenvalid in the last 60 days, set Action if Unavailable to “Queue Seen Addresses”. This isthe recommended setting and is the default.

– If yo would like CanIt-Domain-PRO to queue all mail (even if the recipients have notbeen proven valid), set Action if Unavailable to “Queue All Addresses”. Note: Thissetting runs the risk of causing backscatter and is not recommended.

• Click Submit Changes

To delete a domain/server pair from the table, enable the appropriate Delete checkbox and click Sub-mit Changes.

If you enter a string in the “Filter:” box, then CanIt-Domain-PRO limits the display to entries whoseDomain or Server columns contain that string.

If your verification server listens on a non-standard port (that is, a port other than port 25), youmay specify the port number by following the server name with a slash and the number. For ex-ample, if you have a server called mail.example.com that listens on port 2525, you can usemail.example.com/2525 in the Server column.

Note: If you use a verification server, ensure that the server does not throttle or rate-limit the CanIt-Domain-PRO server in any way. Because CanIt-Domain-PRO runs an SMTP connection for each RCPT com-mand, some naive SMTP server software may think it’s under attack and rate-limit the CanIt-Domain-PRO server, with disastrous results.

5.4.1 Wildcard Verification Server

You may optionally choose to add a Verification Server entry for the wildcard domain of ’*’. This willcause mail for any domain that does not have a specific entry to be checked against that server.

Note: If you are relaying outbound mail via your CanIt-Domain-PRO server, you should NOT use a wild-card verification entry, as it will likely result in the rejection of all outbound mail. You can avoid this



problem by forcing outbound mail through a different realm than inbound mail; in this way, the in-bound realm’s verification server settings are not used for outbound mail. The outbound realm may bea subrealm of the inbound realm if you wish to provide administrative access to the inbound realm’sadministrator.

Finally, note that Verification Server lookups are made only on CanIt-Domain-PRO cluster membersthat are marked “Inbound” in the Cluster Members Table. (Normally, all CanIt-Domain-PRO nodesare marked “Inbound”.)

5.4.2 SRS and Verification Servers

CanIt-Domain-PRO will rewrite the envelope sender using SRS before checking against a VerificationServer if all of the following conditions are met:

1. SRS has been configured.

2. The sending address SPF lookup resulted in “pass”.

3. The quarantine setting “Enable SRS” is set to true in the default stream of the recipient’s realm.The reason CanIt-Domain-PRO looks in the default stream is that address-to-stream mappingis normally only done after a recipient address has been verified.

5.5 Mail Routing

Note: This section is applicable only to Hosted CanIt, to Debian-based CanIt-Domain-PRO appliances or toRed Hat installations with the appliance RPMs installed. On other CanIt-Domain-PRO installations,you need to configure routing manually by editing Sendmail’s access and mailertable files.

Please note the following important requirement:

All of the features in Sections 5.5 through 5.6 rely on SSH to operate. Your system must be runningan SSH server listening on port 22 and it must allow public-key authentication. If you are running acluster, all cluster members must be running an SSH server on port 22 and permit connections fromall other cluster members. If your SSH server listens on a different port, the features will not work.

To configure mail routing, click on Setup and then Domain Routing. The Domain Routing pagecomes up:

Figure 5.6: Domain Routing Screen


5.5. MAIL ROUTING 61

Note that the Domain Routing page shows the routing for all domains in the current realm and in allof its subrealms.

To add a domain for routing:

1. Enter the domain name in the “Domain” box.

2. Click Add Domain

The Domain Routing Detail screen will come up:

Figure 5.7: Domain Routing Detail

1. Enter the servers to which mail should be routed for the given domain. You can enter more thanone server; if you need more than one, enter them one per line. The servers are tried in order,until one successfully accepts or permanently rejects the mail.

2. If you wish the routing server(s) to be treated as MX records, set Treat route entries as MXrecords to Yes. Otherwise, leave it at No.

Note: You should normally not treat your route entries as MX records. Unless you know for sure thatthey specify correct MX records that will route your mail correctly, setting this setting to Yescould cause mail loops. If you use IP addresses rather than host names for your routes, you mustnot set Treat route entries as MX records to Yes.

3. If you wish to route mail to a non-standard port (normally, SMTP traffic goes to TCP port 25),enter the port number in the Destination TCP port box. Note that only the CanIt-Domain-PROsite administrator can specify a port that is less than 1024 and that is not 25 or 587.

Note: CanIt-Domain-PRO imposes a system-wide limit of 12 different non-standard TCP ports. Thislimit is caused by technical limitations in Sendmail and cannot be raised. Again due to Sendmailtechnical limitations, if you specify more than one server in the Route To list, all servers mustlisten on the same port.

Note: If you use a non-standard port for mail routing and are using the Verification Servers feature tovalidate recipients, you may need to specify the same non-standard port in Setup : VerificationServers.



4. CanIt-Domain-PRO can send an alert when either the number of queued messages or the age ofthe oldest queued message exceeds a threshold. The Number of queued messages requiredto trigger notification and Age of queued message in hours required to trigger notificationsettings control when warnings are sent. In order to have alerts generated, you must enter anemail address in the Notification Email Address field, and this address cannot be in the samedomain as the domain being routed. (If mail for example.org is backing up in the queue, itis probably pointless to attempt to mail an alert to someone in that domain.)

If you wish to send an alert to more than one recipient address, enter a comma-separated list ofemail addresses. For example:

[email protected], [email protected]


5.5.1 Outbound Relaying

Normally, CanIt-Domain-PRO refuses to relay mail for domains that do not appear in the DomainRouting Screen. However, if you wish to relay outbound mail through CanIt-Domain-PRO, you canspecify networks for which relaying should be enabled. To do this:

1. Click on Setup and then Known Networks

2. Enter the network from which relaying should be allowed. For example, to allow all machineson the Class C network 192.168.2.0 to relay outbound mail, enter 192.168.2.0/24

3. Enable the Allow Relaying checkbox.

4. Click on Submit Changes.

5.5.2 Outbound Relaying for Select Domains

Normally, CanIt-Domain-PRO enables the Relay Unlisted Domains (rud) flag for a Known Network.This means that if Allow Relaying is enabled, then mail from the given network is relayed regardlessof the sending domain.

If you wish to relay mail from a network only for specific domains, perform the following steps:

1. Click on the Show button in the Associated Domains column corresponding to the appropriateKnown Network.

2. Enter a domain in the “Add Domain” text box and click Submit Changes

3. Repeat the previous step for all sending domains that should be relayed from the given network.

4. Disable the Relay Unlisted Domains flag.


5.6. CLUSTER MANAGEMENT 63

5.6 Cluster Management

The CanIt-Domain-PRO Web interface has a page for managing your CanIt Cluster. To access thepage, click on Setup and then Cluster Management. The Cluster Management page appears:

Figure 5.8: Cluster Management Page

The various machines in your cluster are shown. Each member of a CanIt-Domain-PRO cluster canrun one or more services. The services are:

• Scanner – this service scans mail flowing through the cluster member. Typically, all membersof a CanIt-Domain-PRO cluster will run this service, although large installations may not run ascanner on the database host. NOTE: All nodes should be marked “Scanner” even if they don’tactually act as MX hosts. This is to permit locally-generated traffic such as cron messages to bedelivered. Do not turn off the “Scanner” service on any cluster members. If you think you needto, please contact Roaring Penguin support first.

• Ticker – this service runs periodic maintenance jobs. Exactly one host in the cluster must runthis service. That host must also run the Scanner service.

• Main Database – this service is the main PostgreSQL database. One host in the cluster mustbe an active database server, but it is possible to set up a failover database server.

• Web Server – this service provides the Web interface and REST-based API. it can run on asmany hosts as you like.

• Inbound – this host processes inbound email. Always leave this setting enabled; if you thinkyou need to disable it, please contact Roaring Penguin technical support. Note that the tickermust be marked as an Inbound scanner.

If a host is not marked as inbound, then:

1. Verification Server checks are skipped.

2. (Appliance Only) Domain Routing entries are ignored by the host (mail is routed solelyaccording to MX records) and Sendmail access entries are not created for domains in theDomain Routing table.

• Outbound – this host processes outbound email. Always leave this setting enabled; if you thinkyou need to disable it, please contact Roaring Penguin technical support.

If a host is not marked as outbound, then:

1. The “Allow Relaying” Known Networks flag is ignored.



2. The “Force to Stream” Known Networks entry is ignored.

• Sync Bayes – this host requires Bayes data for processing email. Always leave this settingenabled; if you think you need to disable it, please contact Roaring Penguin technical support.

• Log Host – this host contains mail logs that should be indexed for the Log Search Feature. Notethat this feature is available only on Hosted CanIt and our CanIt appliances.

• Storage Manager – if you are using the Storage Manager, the table will indicate on which hostsit is running. You can run Storage Manager on as many hosts as you like.

5.6.1 Bandwidth Optimization for Copying Files

CanIt-Domain-PRO copies files from the ticker to other cluster members on a regular basis. Forexample, this is how Bayes databases are distributed. If every cluster member is given a non-blanklocation, then CanIt-Domain-PRO can optimize the use of relatively slow links. Here is an example:

Suppose you have three data centres A, B and C. Suppose that within a data centre, cluster membersare connected by 1Gb/s Ethernet, but between data centres there is only a 10 Mb/s link. Furthermore,suppose that you have three hosts in each data centre with the ticker host in A.

If you set the locations of the hosts to “A”, “B”, and “C” according to which data centre they are in,then when CanIt-Domain-PRO copies files, it performs the following steps:

1. The ticker copies the files to all machines in its location (A) and to one machine into each of theother locations. These other machines are called the representatives.

2. Then for each representative, CanIt-Domain-PRO copies the files from that machine to the othermachines that are in the same location as the representative.

You can use whatever labels you like for the Location field as long as machines that are in the samelocation have the same label. Note also that every machine in the cluster must have a non-blanklocation or CanIt-Domain-PRO will not perform bandwidth optimization.

5.6.2 Altering Services on a Cluster Member

To alter the services running on a cluster member:

1. Check or uncheck the appropriate checkbuttons or radio buttons in the Scanner, Ticker, etc.columns. Note that that the Database and Web Server checkboxes are informational; changingthem won’t actually change which services run on the host. And the Storage Manager columnis read-only because storage manager hosts are configured in the Storage Manager Wizard.)

2. Click Submit.


5.7. KNOWN NETWORKS 65

5.6.3 Renaming of Cluster Members

If you rename a CanIt-Domain-PRO host, the cluster management software usually picks up on thename change automatically. If, however, nonexistent or dead hosts appear in the Cluster Managementtable, you can delete them. To delete hosts:

1. Enable the appropriate checkboxes in the Delete column.

2. Click Submit.

Internally, CanIt-Domain-PRO identifies hosts with a UUID, which is an identifier that looks some-thing like this:

30829e66-4df8-11e2-95d2-e6dca73e5dae

The UUID of a given CanIt-Domain-PRO host is stored in the file/etc/mail/canit/canit-cluster-member-id. You can find the UUID by runningthis command:

# head -n 1 /etc/mail/canit/canit-cluster-member-id

In the Cluster Management screen, hovering over the host name reveals the UUID of the host. Thiscan help you to decide which to delete in case two identical host names appear.

5.7 Known Networks


CanIt-Domain-PRO allows you to enter a list of “known networks”. These are typically networksthat you control, and for which you wish to alter the normal CanIt-Domain-PRO processing flow.For example, you may not wish to scan outgoing mail for spam; if all outgoing mail originates from aknown set of IP addresses, you can tell CanIt-Domain-PRO to skip spam-scanning for mail originatingfrom those IP addresses.

To edit the list of known networks, click on Setup and then Known Networks. The Known Networkspage appears:

Figure 5.9: Known Networks

Each network appears as a row in the table. By default, CanIt-Domain-PRO abbreviates the attributenames to avoid a very wide page that requires horizontal scrolling. You can hover over the abbreviationto see the full attribute name, or click Full Headings to show the full attribute names.

In the example in Figure 5.9:



• The host 192.168.10.6 will not be looked up in any RBL blocklists.

• Mail originating from 192.168.10.6 will not be scanned for spam:

• Mail originating from 192.168.10.6 cannot be blocked. That is, any sender, domain or hostblock rules will be ignored.

• Greylisting will be turned off for 192.168.10.6.

• 192.168.10.6 will never be banned by the Dictionary Attack Detector.

• Mail originating from 192.168.10.6 will be streamed into the Outgoing stream, no matter what.

To add a network to the list of known networks:

1. Enter the network address in the Network box. A network address can either be a single IPaddress, or a network address in CIDR notation: a.b.c.d/bits. In this notation, a throughd are decimal numbers from 0 to 255, and bits is a number from 1 to 32 specifying howmany bits of the address are significant. Note that the remaining bits (32 – bits) must bezero. (For more information on CIDR notation, please see http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing.)

Here are examples of network addresses:

• 192.168.1.0/24 specifies the Class C network 192.168.1.0 through 192.168.1.255.

• 10.5.2.0/23 specifies the IP addresses 10.5.2.0 through 10.5.3.255.

• 192.168.5.5/24 is invalid, because the lower 8 bits of the address must be zero.

2. Choose the characteristics you wish to apply to hosts in the known network (you may need toclick on Full Headings to see the full names of each characteristic.)

• To skip DNS-based RBL lookups, enable Skip RBL Lookups (srl).• To skip spam-scanning, enable Skip Spam Scan (sss).• To skip virus-scanning, enable Skip Virus Scan (svs).• To skip filename and filename extension checking, enable Skip Extension Rules (ser).• To skip MIME-type checking, enable Skip MIME-Type Rules (smr).• To skip enforcement by CanIt-Domain-PRO of maximum message size, enable Skip Size

Limit Checks (ssl).• To prevent sender, domain or host block rules from applying to mail sent from the network,

enable Prohibit Block Rules (pb).• To skip greylisting for hosts in the network, enable Skip Greylisting (sg).• To skip SPF checks for hosts in the network, enable Skip SPF Checks (ssc). Note that

this also disables DKIM and DMARC checking.

• To disable delay rules for hosts in the network, enable Skip Delay Rules (sdr).• To disable attachment-stripping rules for hosts in the network, enable Skip Attachment

Stripping (sas).


http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing

http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing

5.7. KNOWN NETWORKS 67

• To prevent any hosts in the network from being banned by the Dictionary Attack Detector(Section 5.3.3), enable Omit from Dictionary Attack Detection (oda).

• If all hosts in the network are “friendly”, then enable Friendly Host (fh). If mail froma friendly host must be rejected, then CanIt-Domain-PRO simply discards it rather thanreplying with an SMTP 5xx code. This is used to prevent backscatter.

• To have CanIt-Domain-PRO parse Received: headers to find the sending relay, enableParse Received Headers (prh). CanIt-Domain-PRO parses through the headers until itfinds a host that isn’t in a known-network with this flag set. If CanIt-Domain-PRO parsesthe Received: headers, then the host that directly initiated the SMTP connection to theCanIt-Domain-PRO scanner is called the Connecting Relay whereas the host parsed out ofthe Received: headers is called the Sending Relay. If CanIt-Domain-PRO does not parsethe Received: headers, then the Sending Relay and the Connecting Relay are one and thesame.

• To auto-allow-always recipients of messages from a known network, enable Auto-AllowRecipients (aar). This means that for messages originating from the network, the recipi-ents of the message are allowed-always in the Sender Rule table.Note that auto-allowing is not applied if any of these conditions holds:

– There is already a sender rule for the recipient in the stream in which the SenderAllow rule would normally be created.

– The message has a “Precedence: bulk” or “Precedence: junk” header.– The message has an “Auto-Submitted” header, as specified in RFC 3834.– The message is a bounce message (in other words, the sender is <>.– The message subject contains “[no-whitelist]”. In this case, the [no-whitelist] tag is

removed before the message is delivered (so that the recipients do not see it.)– The message subject matches the regular expression ˆout of.*office case-

insensitively.– Auto-allowing has been disabled under Preferences : Quarantine Settings for the

sender’s stream.

Note that some auto-responder software ignores RFC 3834 and fails to add an “Auto-Submitted” header. This could lead to situations in which CanIt-Domain-PRO auto-allowssomeone because of an auto-response. If you cannot convince your auto-responder soft-ware to add an Auto-Submitted header, you should complain to the vendor of that softwarein an attempt to make it RFC-compliant.If a stream inherits from a final stream, then the allow-always rule is created in the finalstream. Otherwise, it is created in the actual stream itself. Please see Section 10.3.1 onpage 175 for the precise definition of a final stream.

• To allow outbound mail from the network to be relayed through the CanIt-Domain-PROmachine, enable Allow Relaying (ar).

Note: Outbound relaying can be enabled from the Web interface only on CanIt-Domain-PROappliances or Linux-based RPM builds with the appliance RPMs installed. Also, this flagis ignored on nodes that are not marked “Outbound” in the Cluster Members Table.



• To rate-limit how many recipients per hour a given sender can send to, enter a number inPer-Sender Recipient Rate Limit. If you use this option, you must also enter a Force ToStream value. Rate-limiting is described more fully in Section 5.8.

• If you wish to rate-limit by sending IP address as well as sending email address, enter theappropriate limit in the Per-IP Recipient Rate Limit box. See Section 5.8.1 for details.

• To force all mail from the network to be streamed into a specific stream, enter the nameof the stream in the Force To Stream box. Note: You must supply a fully-qualifiedstream name of the form realm:stream. If you use the magic value @@ as therealm name, then the message is forced into the realm of the envelope sender and thegiven stream. For example, if you set the Force To Stream value to @@:outgoingand the domain example.com is mapped to the realm example-com, then mailfrom [email protected] originating from the known network will be forced into thestream outgoing in the realm example-com. Alternatively, you can use a forced-tostream name of the form somerealm:@@. Continuing our example, that would forcemail from [email protected] originating from the known network into the streamexample-com in the realm somerealm.Note that the Force To Stream box is ignored on nodes that are not marked “Outbound” inthe Cluster Members Table.If mail is forced to a stream, CanIt-Domain-PRO does not perform any SPF, DKIM orDMARC checks, since forcing mail to a stream typically indicates outbound mail.

3. Click Submit Changes to have your changes take effect.

To edit an existing known network, simply adjust the attributes as required and click Submit Changes.To delete a known network, enable the Delete? checkbox and click Submit Changes.

5.7.1 Associating Domains with Known Networks

Each Known Network may be associated with any number of domains. To view the list of associateddomains for a given network, click on the “Show” button in the “Associate Domains” column. Thelist of Associated Domains appears:

Figure 5.10: Known Network with Associated Domains


5.8. RATE-LIMITING OUTBOUND MAIL 69

In this example, the domains example.com and example2.net are associated with the network192.168.7.88. Additionally, email originating from that network is normally forced into the out-bound stream, but if email originating from that network has an envelope sender whose domain isexample2.net, then it will be forced into the outbound-example2.netTo associate a domain with a network, simply enter the new domain name in the Domain box. Youmay optionally specify a domain-specific Force To Stream value in the Force To Stream box; thisoverrides the general Force To Stream setting associated with the network.

Click Submit Changes to add the domain. To remove domains from the list of associated domains,enable the appropriate checkboxes in the Delete? column and click Submit Changes.

If you have enabled the Allow Relaying (ar) flag on a known network, the Relay Unlisted Domains(rud) flag will normally be on as well. This tells CanIt-Domain-PRO to relay all mail from thespecified network, regardless of the sender domain. If, however, you turn off the Relay UnlistedDomains (rud) flag, then CanIt-Domain-PRO will refuse to relay mail from the given network unlessthe domain of the envelope sender is in the list of domains associated with the network. (CanIt-Domain-PRO always permits relaying of the null return path, <>.) We do not recommend turning offRelay Unlisted Domains unless you are absolutely sure the network never originates or forwards mailfrom a domain not in the list of associated domains.

5.7.2 Overlapping Networks

If you add two networks that overlap, CanIt-Domain-PRO will use the most-specific networkfor a given host. That is, CanIt-Domain-PRO will choose the smallest network that con-tains a given host. For example, if you create the known networks 192.168.1.0/24 and192.168.1.240/28, then hosts in the range 192.168.1.240 through 192.168.1.255will use the 192.168.1.240/28 settings, whereas hosts from 192.168.1.0 through192.168.1.239 will use the 192.168.1.0/24 settings.

Note: Because of how Sendmail’s access map works, the handling of overlapping networks described abovedoes not apply to the Allow Relaying (ar) setting. Instead, relaying will be permitted for any host in anetwork with the flag enabled even if there is a more-specific network with the flag turned off. If thisis of concern, then you need to split your Known Networks entries into non-overlapping networks.

5.7.3 The SMTP-AUTH Pseudo-Network

CanIt-Domain-PRO supports a pseudo-network called SMTP-AUTH. (It must be entered exactly likethat in upper-case.) Any Known Network settings for this network will be applied to users whoauthenticate using SMTP AUTH. This lets you do things like force authenticated mail into a particularstream or skip spam-scanning for authenticated users.

5.8 Rate-Limiting Outbound Mail




The Known Networks feature allows you to limit the number of recipients a given sender can mail inan hour. This can be useful to catch compromised internal hosts that are used to send spam. Here ishow rate-limiting works:

• Normally, you can only rate-limit mail from a Known Network. This is because rate-limiting isdesigned to rate-limit outbound mail from a set of machines under your control. Under specialcircumstances, you can enable rate-limiting for any stream other than default, but you shouldnormally not use rate-limiting for inbound email.

• To specify a rate limit, enter the maximum number of recipients per hour that a given sender cansend to. A reasonable value might be 500 to 1000; a value of 0 disables rate-limiting completely.(Enter the value in the Recipient Rate Limit column of Known Networks.)

• The rate limit may be positive or negative. CanIt-Domain-PRO treats limits as follows:

– If the limit is positive, then a sender who exceeds the limit is permanently blocked. Anymail from that sender is rejected with an SMTP permanent-failure code.

– If the limit is negative, then a sender who exceeds the absolute value of the limit is temp-failed. Any mail from that sender is rejected with an SMTP temporary-failure code.

• You must also specify a Force To Stream value in order to use rate-limiting.

If a sender exceeds the rate limit, CanIt-Domain-PRO creates a Sender rule in the Force To Streamstream. The rule rejects all mail from the sender. This has the effect of completely disabling alloutbound mail from the sender address. The sender rule that CanIt-Domain-PRO creates is set toexpire automatically three days after it is created.

CanIt-Domain-PRO also sends an email to the CanIt-Domain-PRO administrator informing him orher of the rule that blocks the sender. Note that the sender will be unable to send outbound mail untilthe administrator goes into the Force To Stream stream and manually removes the rule that blocksthe sender (or until the rule expires after three days.)

Note: Any sender that has any Sender Rule defined in the outbound stream will not be subject to rate-limiting. You can use this as an “escape hatch” to permit certain senders to send high volumes of mail;simply always allow those senders in the forced-to stream (or add a “Hold if looks like spam” rule forthose senders.) However, you should be very careful to do this only for legitimate senders who areunlikely to have their accounts hijacked. Also, note that if a sender is allowed for any reason (ie, asender allow rule, domain allow rule or host allow rule), rate-limiting will not apply. For this reason,you should be very judicious about the allow rules you create in the forced-to stream and considersetting up the forced-to stream not to inherit from the default stream.

Note: If you enable rate-limiting on a Known Network, be sure that you do not enable the “Prohibit BlockRules” option for that network. Otherwise, rate-limiting rules will be ignored! In addition, if yourate-limit the SMTP-AUTH pseudo-network, be sure not to enable the global setting “Always allowusers who use SMTP authentication” (G-3600) or rate-limiting will be ignored.



5.8.1 Rate-Limiting by IP Address

Normally, CanIt-Domain-PRO applies rate-limiting on a per-sender email address basis. If you enablethe Per-IP Recipient Rate Limit feature in Known Networks, CanIt-Domain-PRO will also applyrate-limiting to the sending IP address. If the Known Networks entry has Parse Received Headersenabled, then the IP address that is rate-limited is extracted from the Received: headers.

As with the sender rate-limit, the IP-based rate limit may be positive or negative, with positive limitsyielding an SMTP permanent-failure code and negative ones yielding a temporary-failure code if thelimit is exceeded.

Note: Be very careful when enabling IP-based rate-limiting. If all of your mail goes out through one serverand you accidentally turn on rate-limiting by IP address without enabling Received: header parsing,you may end up blocking all outbound mail. The rule of thumb is as follows:

• If various clients connect directly to the CanIt-Domain-PRO server to send outbound email,you must not enable Parse Received Headers on the Known Network containing the client IPaddresses.

• If clients relay via an SMTP server that subsequently relays out via the CanIt-Domain-PROserver, then you must enable Parse Received Headers.

5.8.2 Fine-Grained Rate-Limiting Rules

Note: By default, realm administrators do not have permission to create rate-limiting rules, but permissioncan be granted by the CanIt-Domain-PRO site administrator.

In addition to per-known-network rate-limits, you can create finer-grained rate-limiting rules by click-ing Rules : Rate Limiting. The Rate-Limiting Rules page appears (Figure 5.11):

Figure 5.11: Rate-Limiting Rules

Normally, CanIt-Domain-PRO only applies rate-limiting rules for a stream if the mail has been forcedinto that stream by a Known Networks match. However, for streams other than default you can



change the setting “Apply rate-limiting rules in stream?” to Always to always apply the fine-grainedrate-limiting rules, even if mail was not forced into the stream by a Known Networks match. Wedo not recommend applying rate-limiting to inbound streams; you should normally never change thissetting.

Rate-limiting rules permit you to use one of the following in the Originator column:

For sending email addresses:

• A full email address, which applies to a specific sender.

• A domain name, which applies to all senders in that domain. Note that a full email address rulewill override a domain rule.

• A single asterisk, which applies to senders that don’t have a full email address or a domain namematch.

For sending domains:

• A domain name prefixed by ‘@’ which limits mail from all senders within that domain.

• The value @* which applies to all domains.

The difference between a sending email address limit and a domain limit is that domain limits applycumulatively to any email address within the domain. Thus, a limit of 100 recipients per hour forexample.com limits any given sender within the “example.com” domain to 100 recipients per hour.On the other hand, a limit of 100 recipients per hour for @example.com limits the total number ofrecipients for all addresses within the “example.com” domain to 100 recipients per hour.

For sending machines:

• An IPv4 or IPv6 address, which applies IP-based rate-limiting to a specific IP address.

• The IP address 0.0.0.0, which applies IP-based rate-limiting to machines that don’t have a spe-cific IP address rule. (This includes IPv6 machines.)

In the example in Figure 5.11, the following sender rate limits apply:

• The sender “[email protected]” is limited to 100 recipients per hour.

• The sender “[email protected]” is limited to 500 recipients per hour.

• The sender “[email protected]” has no rate-limits set.

• All senders in the “example.com” domain are limited to 200 recipients per hour.

• All other senders are limited to 150 recipients per hour by the “*” entry.

The following domain-based rate limits apply:



• Senders in the domain “example.net” are cumulatively limited to 300 recipients per hour.

• Senders in all other domains are cumulatively limited to 200 recipients per hour.

And finally, the following IP-based rate-limits apply:

• The machine 10.2.3.4 is allowed to send to 10000 recipients per hour.

• All other machines are limited to 500 recipients per hour by the “0.0.0.0” entry.

To create a rate-limiting rule:

1. Enter the sender address, domain name, IP address, “*” or “0.0.0.0” in the Originator box.

2. Enter a number from 0 to 100000 in the Hourly Limit box. An entry of 0 means that norate-limiting is to be applied. Any other entry N applies a rate-limit of N recipients per hour.

3. Select an action from the Action pull-down. Available actions are:

• Reject — if the rate-limit is exceeded, CanIt-Domain-PRO creates a rule that blocks thesender or IP address. Mail from the blocked originator will simply be rejected.

• Tempfail — if the rate-limit is exceeded, CanIt-Domain-PRO creates a rule that alwaystempfails the originator. This permits administrators to examine the situation and unblockthe originator if necessary.

• Hold Always — if the rate-limit is exceeded, CanIt-Domain-PRO quarantines all mailfrom the originator. Again, this permits administrators to examine the situation and releasethe quarantined messages if they are legitimate.

4. Enter a number from 1 to 30 in the Block Duration field. When CanIt-Domain-PRO creates aReject, Tempfail or Hold Always rule, it sets it up to expire after N days, where N is the numberyou enter for Block Duration.

5. If you wish, enter a comment in the Comment box to help remind you why you made the rule.

6. Click Submit Changes

To delete rate-limiting rules, enable the appropriate checkbox in the Delete? column and click SubmitChanges.

5.8.3 Notes about Rate-Limiting Rules

• Rate-limiting rules are applied only for mail that is forced into a stream by a Known Networksentry. Normal inbound mail is never rate-limited.

• Rate-limit settings are inherited across streams. CanIt-Domain-PRO uses the best match in themost-specific stream to determine the rate-limit. For example, suppose the stream outboundinherits from the stream default. Suppose that outbound has a rule for “example.com” and thatdefault has rules for “[email protected]” and “*”. Then:



1. An originator “[email protected]” will use the “example.com” entry from outbound.That’s because outbound is more specific than default and it did have an entry thatmatched the originator.

2. An originator “[email protected]” will use the “*” entry from default because no rule inoutbound matched.

• When a rate-limit is hit and a rule is created, the rule is always created in the forced-to streamfrom the Known Networks entry. Additionally, CanIt-Domain-PRO sends an email to the siteadministrator informing him or her that the originator has exceeded the rate limit.

• If you use a Hold Always rule, make sure the forced-to stream is not a tag-only stream. Other-wise, mail from the originator will be tagged rather than quarantined.

• Make sure the forced-to stream is not opted-out of spam-scanning or any hold, tempfail or rejectrules will be ignored.

• If an originator does not match any rate-limiting rules, then the rate limits from the KnownNetwork entry (if any) apply.

• The “Hourly Limit” refers to the total number of recipients mailed, not the number of uniquerecipients. For example, if a given sender sends 50 copies of a message all to the same recipient,that counts as 50 recipients, not one recipient.

5.9 Features


The Features page allows you to globally disable certain CanIt-Domain-PRO features to reduce thenumber of database queries. Note that disabling a feature completely disables it system-wide. Unlessyou know for sure that you don’t need a feature, and you know that the load savings will be worthturning it off, you should leave all features in their default states.

To disable a set of features, click on No in the Enabled column for the features you want to disable.Then click Submit Changes.

Some features are disabled by default because they are considered dangerous or are only useful inspecial situations. You can enable such features by selecting Yes in the Enabled column and thenclicking Submit Changes.

5.9.1 Direct Queue Injection

Normally, when CanIt-Domain-PRO needs to split an incoming message destined for several streamsinto several single-stream messages, it performs the following actions:

1. It remails a copy of the message for each stream by invoking sendmail with appropriatearguments.

2. It discards the original message.


5.10. SYSTEM CHECK 75

Remailing a message with Sendmail is expensive because multiple copies of the message data aremade and Sendmail uses expensive disk synchronization operations after each copy.

CanIt-Domain-PRO can instead directly inject copies of the streamed messages into Sendmail’s localclient queue. This saves disk I/O because only one expensive synchronization operation is needed (notone per copy.) Also, the data can be hard-linked instead of copied, saving disk space.

In order for this to work, the defang user must be a member of the smmsp group. (This is the caseif you are running an appliance or an RPM build.) Additionally, you must enable the “Insert StreamedMail Directly Into Sendmail Queue” feature under Setup : Features.

5.10 System Check

The System Check page runs some sanity checks on your CanIt-Domain-PRO installation. It alsodisplays the current versions of RPTN data and Roaring Penguin rule sets. A typical System Checkpage is shown in Figure 5.12:



Figure 5.12: System Check

In addition to running a few local tests, viewing the System Check page also shows the results ofcluster-wide tests performed on a periodic basis. If System Check indicates a problem, you shouldtake action to fix it immediately. The various System Check tests are outlined in Appendix L.

5.11 Templates

CanIt-Domain-PRO uses templates to configure how Bayes training information is added to messagesand to configure the appearance of Pending Message Notifications. These templates may be configuredon a per-realm basis.

To configure templates, click on Setup and then Templates. The Templates screen appears:


5.11. TEMPLATES 77

Figure 5.13: Templates

The various templates you can configure are:

• Base URL of CanIt installation is used to construct URLs in messages sent out by CanIt-Domain-PRO.

• Base URL for URL-Rewriting is used to construct URLs when rewriting URLs (Chapter 14).Normally, you should leave this template blank, in which case the Base URL of CanIt instal-lation is used.

• E-Mail address of CanIt System Administrator is the e-mail address to which CanIt-Domain-PRO sends certain warning messages or alerts.

• Source E-Mail address of CanIt notifications is the sender address used by CanIt-Domain-PRO when it e-mails notifications. This is the envelope sender address.

• Full name for sender of CanIt notifications is the full name placed in the From: header ofCanIt-Domain-PRO notifications.

• Header From: address of sender of CanIt notifications is the email address placed in theFrom: header of CanIt-Domain-PRO notifications. If this template is left blank, then CanIt-Domain-PRO uses the value from “Source E-Mail address of CanIt notifications”.

• SMTP reply for a rejected incident is the text returned with the SMTP permanent failure codewhen CanIt-Domain-PRO rejects an incident.

• SMTP reply for a blocked entry is the text returned with the SMTP permanent failure codewhen CanIt-Domain-PRO rejects a host, sender or domain that is blocked.



• Header note for an always-allowed entry is the note CanIt-Domain-PRO places in the X-Spam-Score header when a host, sender or domain has an always-allow rule.

• Plain-text training link body specifies the appearance of Bayesian training links added toplain-text messages.

• HTML training link body specifies the appearance of Bayesian training links added to HTMLmessages.

• Pending notification e-mail subject specifies the subject to put in Pending Notification mes-sages.

• Pending notification e-mail body specifies the body of Pending Notification messages

• Preamble before notification details specifies the preamble before the detailed list of heldmessages (for users who select verbose notifications.)

• Detailed pending notification entry specifies the format for each held message in detailednotifications.

• Subject for Add Alternate Address e-mail specifies the subject of the confirmation e-mailsent when someone attempts to add an Alternate Address to his/her stream.

• Body for Add Alternate Address e-mail is the body of the confirmation e-mail describedabove.

• Header for ’Webform’-style Pending Notification is the HTML preamble used for “Web-form” pending notifications.

• Footer for ’Webform’-style Pending Notification is the HTML postamble used for “Web-form” pending notifications.

• Subject line for Periodic Reports is the subject used by CanIt-Domain-PRO when mailing outperiodic reports.

• Body of Periodic Report e-mail is the body used by CanIt-Domain-PRO when mailing outperiodic reports. It should consist of valid HTML.

• Text boilerplate when attachments are stripped is appended to the first text/plain emailpart if an attachment is stripped and stored on the CanIt-Domain-PRO server.

• HTML boilerplate when attachments are stripped is appended to the first text/htmlemail part if an attachment is stripped and stored on the CanIt-Domain-PRO server.

• Text boilerplate when attachments are discarded is appended to the first text/plainemail part if an attachment is stripped and discarded.

• HTML boilerplate when attachments are discarded is appended to the first text/htmlemail part if an attachment is stripped and discarded.

• Forgot-your-Password Link or Text is the link or text used for the Forgot your Password?message on the login page.


5.12. THEME CUSTOMIZATION AND BRANDING 79

• HTML content for anti-phishing URL Redirection page is the content of the URL Proxywarning message. See Chapter 14, “URL Proxying”, for details on the URL Proxying feature.

• HTML content for anti-phishing URL Redirection page encountering a Phishing URL isthe content of the URL Proxy message when a suspected phishing link is encountered. SeeChapter 14, “URL Proxying”, for details on the URL Proxying feature.

Note that many templates include various “replacement tags”. For example, in the training link tem-plates, the sequence of characters %spamurl or %{spamurl} will be replaced with a URL thatvotes the message as spam. To see the list of available replacement tags, click on the “(Tags)” linknear the template entry box.

If you change the value of a template in a non-base realm, you can revert to the previous value byclicking the “(Revert to Original)” link next to the template name.

5.12 Theme Customization and Branding

CanIt-Domain-PRO ships with several themes which control the “look and feel” of the Web interface.Some of those themes can be customized. That is, although the basic layout of the theme cannot bechanged via the web interface, the colors of various elements can be and (in some cases) the logo canbe changed as well. This permits you to “brand” CanIt-Domain-PRO with your corporate logo.

To customize a theme, click on Setup : Theme Customization. The Theme Customizations pageappears:

Figure 5.14: Theme Customizations

Note: The list of available customizations is specific to the current theme and realm. If you switch themesor realms, then the list of available customizations will change. Also, some elements such as imageson the login page may only be cusomizable in the base realm and therefore can be customized onlyby the site administrator.

To activate a customization, enable the corresponding Active radio button and click Submit Changes.That customization will become active for the current theme and realm. It will also be active for allsubrealms unless overridden within a subrealm.

To deactivate all customizations, click Deactivate All. This will revert the current theme and realmincluding subrealms to the default un-customized appearance.



To delete a customization, enable the corresponding Delete? checkbox and click Submit Changes.

5.12.1 Creating or Editing a Customization

To add a new customization, click Add New Customization. To edit an existing customization, clickon the name of the customization you wish to edit. In either case, the Theme Customization Editorappears:

Figure 5.15: Theme Customization Editor

The Theme Customization Editor lets you alter the appearance of various components of the webpage. To edit a customization:

1. If you are adding a new customization, the Customization Name field will be blank. Enter thename of your new customization. Note that customization names must be unique for a giventheme and realm.

2. To change image items, upload a GIF, JPEG or PNG file from your computer.

3. To change color items, enter a “#” followed by an HTML color triplet in the text box. If youclick on the color swatch to the right of the text box, you can pick a color from a color selector.

4. Some themes may allow you to enter arbitrary CSS information. This lets you have very finecontrol over the appearance of the theme, but you should not make use of this facility unless youare very familiar with HTML and CSS.

5. If you want to revert a particular item to its theme default, enable the Revert to default? check-box.

6. Click Save to save your customization and continue editing it. Or click Save and return to listto save your customization and return to the list of available customizations.

Note that while you are editing a customization, it becomes active so you can see in real-time whatthe customized theme looks like. Other users, however, will not see the customized theme until youactivate it from the list of customizations.


5.13. HTTPS 81

5.12.2 Emergency Recovery from Bad Theme Customization

If you make a mistake while creating a theme customization and end up with web pages you can’tread or navigate, follow these emergency instructions:

• Look at the URL in the URL bar of your browser. If it contains a question-mark, add thefollowing text on the end of the URL:

&disable theme customization=1

If it does not contain a question mark, add this at the end:

?disable theme customization=1

• Press Enter to visit the newly-edited URL

• Navigate back to Setup : Theme Customization and fix the problem. Note that you have toadjust the URL in the URL bar each time you navigate to a new page, so you might need to doit a few times until the problem is fixed.

5.13 HTTPS

Note: This feature is available only on Debian-based Appliances.

On CanIt-Domain-PRO appliances, HTTPS is enabled by default, but with dummy self-signed certifi-cates. If you would like to install your own certificates, click on Setup : HTTPS. Then:

1. Copy-and-paste your SSL certificate into the first text box. If your certificate provider requiresyou to install an intermediate certificate chain, paste the entire contents of the certificate chainfile into the first text box immediately after you paste in your SSL certificate.

2. Copy-and-paste the corresponding server key into the second text box. The server key must notbe encrypted or the Web server on the appliance will fail to start.

3. Click Submit Changes to install the key and certificate.

5.14 The Domain Mapping Table

Recall from Figure 2.4 on page 34 that CanIt-Domain-PRO uses a Domain Mapping Table to deter-mine how to stream messages for each domain. The table contains a list of domains with a correspond-ing lookup method. To edit the Domain Mapping Table, click on Setup and then Domain Mappings.The Domain Mappings page appears:



Figure 5.16: Domain Mappings

To add a mapping method for a particular domain, enter the domain name in the top row of the tableand select a value in the Mapping column. The possible choices are:

• Database—CanIt-Domain-PRO will look up a stream mapping in the Address Mapping Table(Section 5.15).

• AsIs—CanIt-Domain-PRO converts an address to a stream by removing any angle-brackets andconverting letters to lower-case.

• ChopDomain—CanIt-Domain-PRO converts an address to a stream simply by chopping off [email protected] part, removing any angle-brackets, and converting to lower-case.

• ChopUser—CanIt-Domain-PRO converts an address to a stream simply by chopping off theaddress@ part, leaving just the domain (without angle-brackets and converted to lower-case.)

• Program—CanIt-Domain-PRO converts an address to a stream by executing theaccount-info program. Please see Section 7.2.5 on page 157 for more details. Note thatProgram is deprecated; you should create and use a User Lookup method instead.

• None—CanIt-Domain-PRO removes the domain from the Domain Mapping Table.

• If you have added external User Lookup methods (Chapter 7), some of them may appear asadditional choices. For example, the LDAP, Rewrite and Program User Lookup methods canconvert an address to a stream. If there are any User Lookup methods added to ancestor realmsof the current realm, they will appear as additional choices if they are marked as being availablefor subrealms.

Click Submit Changes to save your changes.

To modify the mapping for an existing domain, select a new mapping in the Mapping column andclick Submit Changes.


5.15. THE ADDRESS MAPPING TABLE 83

Given a domain sub.example.com, CanIt-Domain-PRO looks up entries in the Domain MappingTable in the following order, stopping at the first one found:

1. sub.example.com

2. example.com

3. com

4. *

The special domain * is used as a last resort if no better match is found. You may enter a mapping for* to set a default mapping. If there is no * entry and a domain is not found in the Domain MappingTable, then CanIt-Domain-PRO uses a default lookup method of Database.

If you enter a string in the “Filter:” box, then CanIt-Domain-PRO limits the display to entries whoseDomain or Mapping columns contain that string.

5.15 The Address Mapping Table

CanIt-Domain-PRO uses an Address Mapping Table (Figure 2.4 on page 34) to map e-mail addressesto streams. The Address Mapping Table is used both for hand-entered entries placed there by theCanIt-Domain-PRO administrator, and for caching the results of the Program mapping method.

Note: If there is an exact match for an email address in the Address Mapping Table, then it is always used,overriding any mapping method.

To edit the address mapping table, click on Setup and then Address Mappings. The Address Map-pings page will appear:



Figure 5.17: Address Mappings

To add an entry for a new e-mail address, enter the new address in the Address column of the firstrow, and enter the stream name in the Mapping column. Then click Submit Changes.

To edit an existing entry, edit the text in the Mapping column and click Submit Changes. To deletean entry from the table, click the Delete link in the appropriate row.

Click on Not Cached to see only non-cached (hand-entered) entries, Cached to see only cachedentries, or Any to see all entries in the Address Mapping Table.

If you enter a string in the “Filter:” box, then CanIt-Domain-PRO limits the display to entries whoseAddress or Mapping columns contain that string.

5.15.1 Wild-Card Entries

The address mapping table may contain three types of wildcard entries:

1. The entry user@* is used if CanIt-Domain-PRO is unable to map an address to a stream withan exact match. If you run several domains, but all user-parts are the same, this wildcard can beuseful.

2. The entry *@domain.tld is used if the previous wildcard does not match anything. Use thisentry to set up a default stream for e-mail to a particular domain.

3. The entry * is used as a last resort if the previous wildcards did not match.

Note: The addresses postmaster, postmaster@localhost and postmaster@machine nameare always mapped to the default stream unless you have a specific entry in the Address Mapping


5.16. THE DEFAULT STREAM 85

Table for those addresses. That is, for those three specific addresses, CanIt-Domain-PRO will not usewildcard matches or User Lookups to determine the stream. (In the third address, machine name isthe name of the host processing the email.)

5.16 The default Stream

CanIt-Domain-PRO has a built-in stream name that is reserved, and which cannot be used for otherpurposes. This stream is named default, and is used as follows:

If CanIt-Domain-PRO is unable to map an address to a stream (for example, if there are no exact orwildcard matches in the database and the Program method fails), the address is mapped to the hard-coded stream default. The CanIt-Domain-PRO administrator should check the default streamfrom time to time.

The default stream also contains allow and block rules and custom rules that all other streams caninherit. The factory default is for all streams to inherit the lists and rules from default, but youcan disable this if you wish. List and rule inheritance work as follows for streams that inherit fromdefault:

• Senders, hosts, domains, extension rules and MIME type rules and are first looked up in thestream’s table. If no entry is found, they are looked up in default’s table.

• Custom rules are evaluated first for the given stream, and then for default. Their scores areadded together. Note that if the same rule appears in both the stream’s rule set and default’srule set, it is counted twice.

5.17 Mapping Scenarios

To give a feel for how to use the mapping, we illustrate a few common scenarios.

5.17.1 Central Scanning with Opt-Out

If you run a mail server and wish to centralize spam-scanning, but you have some users who wish toopt out or handle their own spam, you can do it as follows:

In the Address Mapping Table, add this catch-all entry:

Address Stream* admin

This streams most users’ e-mail to the “admin” stream for centralized processing. If [email protected] does not want his mail examined by the spam control officer, simply addanother entry:

Address [email protected] joe

This streams mail for [email protected] to joe.



5.17.2 Single Domain

If you host a single e-mail domain, and each user’s login name is simply the first part of his/her e-mailaddress, setting up mappings is easy. In the Domain Mapping Table, add a single entry:

Domain Mapping Method* ChopDomain

5.17.3 Single Domain with Aliases and Mailing Lists

Most likely, your scenario is more complex than in Section 5.17.2. You probably host mailing lists,and have aliases. Let’s suppose you host a list called [email protected], which is run byjane, and that your [email protected] is an alias which gets expanded to jim and bob.

You can still use the same Domain Mapping as Section 5.17.2. You have two options for handling themailing list and sales alias:

1. Allow jane to access the tv-list stream, and allow jim and bob (or delegate one of them)to access the sales stream. Jane will have to remember to check the tv-list quarantine aswell as her own quarantine, and similarly for Bob and Jim.

2. Add address mappings like this:

Address [email protected] [email protected] bob

Explicit entries in the Address Mapping Table will override even the ChopDomain method.

Here, Jane’s quarantine will contain messages both for herself directly and the mailing list sheruns. Bob’s quarantine will contain his messages and messages for sales. (Clearly, you’vedelegated spam handling for sales to Bob alone.)

(You can, of course, use Method 1 for tv-list and Method 2 for sales. It’s up to you.)

5.18 Pausing Delivery to Selected Domains

Note: This section is applicable only to CanIt appliances or the Hosted CanIt service.

CanIt-Domain-PRO permits you to temporarily pause delivery to selected domains. When deliveryto a domain is paused, CanIt-Domain-PRO will continue to accept mail for that domain, but will notattempt to deliver it to the back-end server. Instead, it will simply queue the mail. When delivery isresumed, the mail will be delivered out of the queue.

5.18.1 Pausing Delivery

To pause delivery to a domain:

1. Click on Setup and then Paused Delivery.


5.19. THE DOMAIN OVERVIEW PAGE 87

2. Enter the domain whose delivery should be paused in the Domain box.

3. Select a Pause Mode. The choices are:

• Delivery and LDAP/Verification: In this mode, CanIt-Domain-PRO will not attempt toconnect to any LDAP servers or verification servers. It will accept mail for addresses thatare in its cache and will tempfail mail for addresses that have not been verified or found inthe LDAP directory recently.

• Delivery Only: In this mode, CanIt-Domain-PRO will not attempt to deliver mail, but willstill connect to LDAP servers and verification servers as usual.

4. Enter the expiry time in the Expiry (Minutes) box. The site administrator may specify up toone day (1440 minutes) and realm administrators may specify up to four hours (240 minutes).


5.18.2 Resuming Delivery

To resume delivery to a domain:

1. Click on Setup and then Paused Delivery.

2. Enable the Delete? checkbox for the domain whose delivery should be resumed.


5.19 The Domain Overview Page

For convenience, CanIt-Domain-PRO allows you to view the most important settings for your domainsin one place. To see the overview, click on Setup and then Domain Overview. If you have more thanone domain mapped to your realm, click on the domain name whose overview you desire. The DomainOverview Page appears:

Figure 5.18: Domain Overview Page

The Domain Overview page shows some or all of the following information:

• The Verification Server settings for the domain. Click Edit to modify the settings.



• The Domain Routing settings for the domain. Again, click Edit to adjust the settings.

• The Domain Mapping and Authentication Mapping settings for the domain. If the domain usesa User Lookup for mapping or authentication, you can click on the link in the Value column tosee the specific user lookup settings.

• Whether or not the domain correctly validates recipients (as checked by the last nightly cronjob.)

• Whether or not the domain’s MX records point at the CanIt-Domain-PRO filter (as checked bythe last nightly cron job.)

5.20 Autotask® Integration

Note: Autotask integration is available only on Hosted CanIt and our Debian-based CanIt-Domain-PROappliances. It is not available in the source or RPM versions of CanIt-Domain-PRO.

Autotask® is a Professional Services Automation package designed for IT consultants and managedservice providers. CanIt-Domain-PRO can interface with Autotask to automatically generate billinginformation so you can invoice your clients on a monthly basis.

The basic workflow for Autotask integration is as follows:

• Within Autotask, set up a product corresponding to CanIt-Domain-PRO services.

• For each customer, set up a realm within CanIt-Domain-PRO and an account within Autotask.

• For each customer, set up a monthly billing contract within Autotask

• Provide enough information to CanIt-Domain-PRO that it can push usage statistics to Autotask.CanIt-Domain-PRO generates or updates a Contract Cost item, thereby permitting automaticinvoice generation.

Once Autotask integration is configured, CanIt-Domain-PRO will automatically post Contact Costs toAutotask with a Unit Quantity corresponding to the number of mailboxes. The Contract Costs will beupdate every day; this means that whatever your billing cycle is, Autotask will always have up-to-dateusage statistics.

5.20.1 Preparing Autotask

To prepare Autotask for CanIt-Domain-PRO integration, perform the following steps within yourAutotask account:

Create a Product corresponding to each CanIt-Domain-PRO service

Under Admin : Features & Settings : Products & Services, create a Product for each CanIt-Domain-PRO service that you offer. Once you have finished, the results will look like Figure 5.19:


5.20. AUTOTASK® INTEGRATION 89

Figure 5.19: Autotask Product List

The possible products are:

• Inbound Scanning. In this example, we called the product CanIt-Inbound.

• Outbound Scanning. In this example, we called the product CanIt-Outbound.

• Secure Messaging. In this example, we called the product CanIt-SecureMessaging.

• Archiving. You should create one product for each possible retention time in months thatyou sell. All of these products must have the same prefix, followed by -n where n is theretention time in months. In Figure 5.19, we created three products with the common prefixCanIt-Archiving and retention times of 1, 12 and 24 months.

Create a Recurring Service Contract within Autotask for each CanIt-Domain-PRO cus-tomer

Under Contracts, create a Recurring Service Contract for each CanIt-Domain-PRO customer. Fig-ure 5.20 shows a sample contract, which we have named Email Security:



Figure 5.20: Autotask Recurring Service Contract

5.20.2 Preparing CanIt-Domain-PRO

To prepare CanIt-Domain-PRO for Autotask integration, log in to your top-level realm (the “base”realm if you are running CanIt-Domain-PRO on-premises or your realm if you are using Hosted CanIt.Click on Setup : Autotask Integration. The Autotask settings screen appears:



Figure 5.21: Autotask Integration Settings

Basic Information

Fill in the basic information needed to integrate with the Autotask API. Note that all settings followrealm inheritance; you can override them as necessary on a per-realm basis. The basic settings are:



• Autotask proxy URL: The URL for accessing the Autotask API. The default value is probablyfine and should not normally need to be changed.

• Autotask API username: A username with permission to access the Autotask API.

• Autotask API password: The password for the API user.

• Account name associated with current realm: The name of the account within Autotask. Thissetting links the current realm in CanIt-Domain-PRO to the account in Autotask.

• Contract name associated with email security product: The Autotask contract name corre-sponding to the CanIt-Domain-PRO services.

• Billing metric: One of Addresses or Streams, depending on whether you bill on the basis ofnumber of email addresses or number of streams.

• Bill for subrealms as well as current realm: If set to Yes, then statistics for the current realmand all of its descendants are counted for billing purposes. If set to No, then only statisticswithin the current realm are pushed to Autotask.

Per-Product Settings

CanIt-Domain-PRO lets you push billing data for up to four product categories to Autotask. The fourcategories are shown below. Note that you may not offer all categories to all of your clients.

1. Inbound Filtering: Inbound email filtering.

2. Outbound Filtering: Outbound email filtering.

3. Secure Messaging: Secure Messaging Service

4. Archiving: Email archiving.

To link each product to Autotask, fill in the following settings:

• Product name: The name of the corresponding product within Autotask. Note: Archiving isa special case because CanIt-Domain-PRO always appends -n where n is the retention timein months. Therefore, in the Archiving Settings section, the Product name setting specifies theprefix to use.

• Unit cost: The unit cost of the product. For most products, this is simply a decimal number.For Archiving, it is a string of the form:

n1=c1,n2=c2,...

which specifies that the cost for n1 months of retention is c1, for n2 is c2 and so on. You shouldenter all combinations of retention time that are actually used by your clients. For example, inFigure 5.21, the Unit cost of 1=0.5,12=1,24=2 means that one month of archiving costs$0.50; 12 months costs $1.00 and 24 months costs $2.00



• Unit cost: The unit price of the product. For most products, this is simply a decimal number.For Archiving, it follows the same format as Unit price.

• Minimum number of units to bill: The minimum number of units to bill each month, if any.For most products, this is an integer, but for Archiving, it follows the same format as Unit priceexcept only integers can appear to the right of each equals sign.

• Contract cost description: If non-blank, the description to use in the contract cost line item. Ifthis is left blank, the description is copied from the Autotask Product.

Once you’ve entered the values for your top-level realm, switch into each customer realm and set(minimally) the Account name associated with the current realm as well as any other settings thatshould be overridden.

5.20.3 Testing the Autotask Integration Settings

You can test the Autotask settings by enabling the Run a live test of these settings against Autotaskcheckbox and clicking Submit Changes. CanIt-Domain-PRO will print a debug log and let you knowwhether or not the settings look correct. Note that because fetching actual statistics is costly, thedebugging output always pretends to post random unit counts to Autotask. In production, the correctnumber of addresses or streams would be posted to Autotask.

If you enable the Push a dummy ContractCost item to Autotask checkbox, then in addition torunning the tests, CanIt-Domain-PRO will push a ContractCost item up to Autotask

A successful debugging log is shown in Figure 5.22.



Figure 5.22: Autotask Test Results


5.21. CONNECTWISE® INTEGRATION 95

Sample Contract Costs as they appear in Autotask after CanIt has pushed statistics are shown inFigure 5.23.

Figure 5.23: Autotask Contract Costs

5.20.4 Autotask Settings and Inheritance

In the Autotask Integration Settings screen, the “Origin Realm” column shows the realm in which asetting has been created. If the setting exists in the current realm, you can check the Reset? checkboxto remove the setting and make the setting once again inherit the value from the parent realm.

Any realm that does not have an Autotask account name associated with it will not have its statisticspushed to Autotask.

Note: The first time CanIt-Domain-PRO connects to Autotask, it extracts the Account, Contract and ProductIDs from Autotask and from then on uses the Autotask IDs rather than the names to link to Autotask.This allows you to rename objects within Autotask without breaking the CanIt-Domain-PRO integra-tion. As a convenience, if CanIt-Domain-PRO notices that an object has been renamed in Autotask, itupdates its copy of the corresponding name to match Autotask’s.

5.21 ConnectWise® Integration

CanIt-Domain-PRO can automatically update mailbox counts in a ConnectWise Agreement Addi-tion. These updates are done nightly, meaning that whenever your billing cycle falls, the AgreementAddition will have up-to-date counts.

5.21.1 Preparing ConnectWise

1. Create a new Product for each CanIt service you offer. The product names must be as follows;you only need to create those products that you are actually using.

• CanIt-Inbound for inbound email filtering.

• CanIt-Outbound for outbound email filtering.

• CanIt-SecureMessaging for secure messaging.

• CanIt-Archiver-n for archiving with a retention time of n months. You need tocreate one CanIt-Archiver-n Product for each retention time you offer.



See Figures 5.24 and 5.25 for examples of how to create the Products within ConnectWise.

Figure 5.24: CanIt-Inbound ConnectWise Product



Figure 5.25: CanIt Product List



2. Set up an Integrator Login ID and Password

Under System : Setup Tables : Integrator Login, create a login for CanIt-Domain-PRO to ac-cess the API. The login that you create must be able to access the following APIs: ManagedServices API, Company API, Product API, Reporting API, System API and Agreement API.See Figure 5.26.

Figure 5.26: Integrator Login ID Setup



3. Create a Management IT Solution for CanIt-Domain-PRO billing. Under System : Setup Ta-bles : Management IT Solution List, create a Management IT Solution. The name must beCanItBilling and the Management IT Solution should be Custom. See Figure 5.27.

Figure 5.27: CanItBilling Management IT Solution Setup



4. Set up Managed Devices Integration. Under System : Setup Tables : Managed Devices In-tegration List, add a CanItBilling entry with the solution set to CanItBilling. TheIntegrator Login should be set to the login name you made in Step 2 earlier. See Figure 5.28.

Figure 5.28: CanItBilling Managed Device Integration Setup



5. Create an Agreement for your customer, if there isn’t one yet. See Figure 5.29.

.

Figure 5.29: Connectwise Agreement



6. Create a new Agreement Addition (if one does not yet exist) for each CanIt product that youwill bill for. See Figure 5.30.

.

Figure 5.30: Connectwise Agreement Addition



5.21.2 Preparing CanIt-Domain-PRO

1. In your main realm, click on Setup and then ConnectWise® Integration. The ConnectWisesetup page appears (Figure 5.31):

Figure 5.31: ConnectWise Setup - Main Realm

In the main realm, all you should enter are:

• ConnectWise Web Site URL: the URL for accessing the ConnectWise API.If you are unsure what it is, please contact your ConnectWise adminis-trator or ConnectWise technical support. If your regular URL is some-thing like na.myconnectwise.net, then the API URL is probablyapi-na.myconnectwise.net/v4 6 release/apis/2.0.

• ConnectWise Company for Login: The company name you use to log into ConnectWise.

• ConnectWise Username for Login: The username you created in Step 2 in Section 5.21.1.

• ConnectWise Password: The password you created in Step 2 in Section 5.21.1.

• Agreement Name: You can fill in a default Agreement Name to use for all of your billingpurposes.

• Billing Metric: Choose Addresses if you are billing based on the number of email ad-dresses, or Streams if based on streams.

• Bill for Subrealms as Well: Set to Yes if you want to bill a realm for its own mailboxesand those of its subrealms, or No if you only want to bill for mailboxes within the specificrealm.



• Bill Customer: This setting allows you to override the BillCustomer flag; select one ofBillable (the default), DoNotBill or NoCharge.

• Unit Price: You may optionally override the unit price by entering a decimal number.

• Unit Cost: You may optionally override the unit cost by entering a decimal number.

You should not fill in anything for the Company Name Associated with this Realm since this isspecific to each customer being billed.

2. For each customer realm that should be billed, switch into that realm and click Setup and thenConnectWise® Integration. In this page, enter the Company Name associated with the realm;it must exactly match the Company name in ConnectWise. You can also override other settingssuch as Agreement Name, Billing Metric, Bill for Subrealms as Well, Bill Customer, Unit Priceand Unit Cost, if necessary. Also, make sure the Agreement Name matches the Agreement youset up in Step 5 in Section 5.21.1.

You should test the connectwise settings by enabling “Run a live test of these settings againstConnectWise” and “Update AgreementAddition data on ConnectWise”. Then click SubmitChanges. If all goes well, the test results will look something like Figure 5.32.

Figure 5.32: ConnectWise Test Results


Chapter 6

CanIt-Domain-PRO Administration

6.1 Global Settings


The first administrative task you should undertake is to set up global settings. Click on the Adminis-tration link. You will see the global settings screen:

Figure 6.1: Global Settings

Note that the Basic Setup Wizard (Section 5.3.1) populates some of these settings. The “ID” columnis a unique identifier for each setting; it is not used except as a convenient way for Roaring Penguinsupport personnel to indicate a particular setting over the phone.

The global settings have the following meanings:

G-1100 Maximum size of message to scan for spam (kB) Spam-scanning can be very slow on large


106 CHAPTER 6. CANIT-DOMAIN-PRO ADMINISTRATION

messages. If a message comes in that is larger than this threshold, CanIt-Domain-PRO attemptsto reduce its size by removing non-text attachments before feeding the message to the scanningengine. If this succeeds, the reduced message is scanned. If the message is still too large evenafter the reduction, it is not scanned for spam.

G-2400 Handling for messages containing viruses If you have a virus-scanner compatible withCanIt-Domain-PRO, this setting controls how CanIt-Domain-PRO deals with virus-bearingmessages. Hold holds the message in the quarantine for approval (or tags the message if thestream is in tag-only mode.) Accept permits the message to pass, while Reject rejects it withan SMTP failure code. Finally, Discard simply discards the message. We recommend settingthis option to Discard.

Note: This setting may be overridden on a per-stream basis.

G-1500 Expire statistics after this many days Once a day, a cron job removes old entries from thestatistics table. By default, CanIt-Domain-PRO keeps statistics for 10,000 days (around 27years), but you can lower this setting to as low as 90 days if you do not want to keep oldstatistics around.

G-1550 Number of hours to keep detailed statistics CanIt-Domain-PRO keeps very detailed statis-tics for a limited time. This setting lets you adjust the length of this time.

G-1600 Expire old data after this many days Once a day, a cron job purges old messages, log entriesand incidents from the database. We recommend retaining at least 14 days’ worth of data,although you might want to lower this on a busy mail server. Note: This setting is the numberof days from the creation of the incidents being expired, regardless of whether or when theywere marked as spam or non-spam.

G-1610 Remember change history for this many days Most CanIt-Domain-PRO web pages have a“Show Changes” link that lets you see changes made to rules and settings. This setting specifieshow long change history should be retained. It may be set to any integer from 45 to 10000 anddefaults to 732 days (about two years).

G-1700 Expire messages marked as spam after this many days This setting controls when the cronjob expires messages you have marked as spam. Note that it only applies to closed incidents—that is, messages that have not only been marked as spam, but have also actually been rejectedby CanIt-Domain-PRO.

G-1800 Expire messages marked as non-spam after this many days This setting controls when thecron job expires messages you have marked as non-spam. Note that it only applies to closedincidents—that is, messages that have not only been marked as non-spam, but have also actuallybeen delivered by CanIt-Domain-PRO.

G-4010 Number of hours to cache address-to-stream lookups As mentioned in Section 2.4,address-to-stream mappings may be cached in the Address Mapping Table. This settingspecifies for how long cached entries remain valid.

G-4015 Number of hours before refreshing cached address-to-stream lookups If a cached addressis older than this many hours, CanIt-Domain-PRO attempts to perform an address-to-stream


6.1. GLOBAL SETTINGS 107

mapping to refresh the cached entry. If the lookup fails with a temporary failure, CanIt-Domain-PRO does not update the cached entry, but will continue to use it until it expires as per settingG-4010. If the lookup succeeds, CanIt-Domain-PRO updates the cached entry. If it fails with a“No such user” result, CanIt-Domain-PRO deletes the cached entry.

G-4050 Time in hours to delay messages with Delayed Attachments If you use the Delayed At-tachments feature, this setting controls the length of the delay.

G-4800 Number of days to keep mail signatures for Bayesian analysis This setting specifies howlong after a message first arrives a user may vote on whether it is spam or non-spam.

G-4900 Number of generations before cleaning common Bayes tokens CanIt-Domain-PRO pe-riodically cleans old data out of the Bayes database. This setting controls how longCanIt-Domain-PRO retains a token that has been seen frequently, but not recently. Werecommend leaving it at the default value.

G-5000 Number of generations before cleaning uncommon Bayes tokens CanIt-Domain-PRO pe-riodically cleans old data out of the Bayes database. This setting controls how long CanIt-Domain-PRO retains a token that has been seen infrequently and not recently. We recommendleaving it at the default value.

G-4020 Users must opt in to anti-spam scanning? If you set this to Yes, then users must explicitlyopt-in to anti-spam scanning. If users do not opt-in, their mail is simply passed through un-changed. If you set this to No, then all users are implicitly opted-in. They can, however,explicitly opt out if they choose.

G-4030 Users must be approved for anti-spam scanning? If you set this to Yes, then the CanIt-Domain-PRO administrator’s approval is required before a user can opt in to anti-spam scan-ning. If you are selling anti-spam scanning as a value-added service, you should set this to Yes.If anti-spam scanning is part of your basic service, set it to No.

Note that opting in and opting out is done on a per-stream basis. Usually, a stream correspondsto a user, but it is possible for a stream to correspond to more than one user, and for a singleuser to be responsible for more than one stream.

G-4300 Minimum size of spam corpus for Bayesian analysis CanIt-Domain-PRO will not useBayes data until at least this many messages have been trained as spam.

G-4400 Minimum size of non-spam corpus for Bayesian analysis CanIt-Domain-PRO will not useBayes data until at least this many messages have been trained as non-spam.

G-3600 Always allow users who use SMTP authentication If your version of Sendmail is compiledto support the SMTP AUTH extension, you can always allow mail from authenticated sendersby setting this to Yes. (The default is No.) In this case, mail from authenticated users will notbe scanned for spam (but will still be scanned for viruses and bad filename extensions or MIMEparts.)

Note: CanIt currently cannot preserve SMTP AUTH-based allow-rules when messages are streamed.Thus, if an AUTH’ed user sends mail to recipients in more than one stream, the allow rule willnot be applied.



G-3900 Store both raw and decoded messages in incident database Some e-mail messages are ob-scured using Base64 encoding or some other encoding scheme. If you change this setting to Yes,CanIt-Domain-PRO stores both the “raw” and “decoded” message in the incident database. Thislets you view encoded messages more reliably, but approximately doubles the disk space usedby the incident database. If you set it to No (the default), CanIt-Domain-PRO stores only theraw message.

The message display Web page can decode some encoded messages, but it is not completelyreliable. If you need a completely reliable way to view encoded messages, you should changethis setting to Yes.

G-4000 Obscure To, Cc and Bcc fields for non-root users Because CanIt-Domain-PRO stores mes-sages that hash identically only once, the To:, Cc: and Bcc: headers of messages may leakrecipient information to other recipients of the message. To hide this information, change thissetting to Yes.

G-4060 Users authenticated by external means default to simple GUI? If you set this to Yes, thenusers who authenticate via an external authentication mechanism have a much simplified inter-face to CanIt-Domain-PRO by default. This simplified interface is described in Chapter 10.

G-4075 Switching to expert mode cancels stream inheritance If you use the Simple Interface(Chapter 10), then you may wish to cancel inheritance whenever a user selects the expert in-terface. In that case, change this setting to Yes. That is, if a user has selected a particularspam-scanning level in the Simple Interface, then when they switch to Expert Interface, the se-lected level is no longer used—instead, individual settings are used that do not depend on anyof the preconfigured spam-scanning settings.

G-4080 Support the Sendmail ‘plus hack’ for streaming Some Sendmail configuration files allowusers to add a “+” sign followed by arbitrary text to their user names, and use the resultinge-mail addresses for various purposes such as filtering e-mail. If you change this setting to Yes,then CanIt-Domain-PRO ignores a “+” sign and any following text after the user name partwhen mapping e-mail addresses to streams.

Note that if you use the “Program” method to stream e-mail, the “+” sign and any following textis retained; it is up to your program to implement the sendmail “plus hack” if you choose.

G-4090 Scan for viruses prior to streaming incoming mail If you know for sure that you alwayswant to reject or discard viruses, regardless of any per-stream settings, then change this settingto Yes. It causes any viruses to be discarded or rejected (according to the global virus-handlingsetting) before any streaming takes place. If a virus comes in for more than one recipient, thiscan greatly reduce the load on CanIt-Domain-PRO. Note that the global virus-handling settingmust not be set to Hold/Tag for this setting to take effect.

G-4100 Timeout in seconds for Verification Server queries If you are using the Verification Serverfeature, CanIt-Domain-PRO will time out Verification Queries according to the value of thissetting. You should keep it reasonably low so that a slow or dead verification server does notinterfere with delivery to other domains.

To make your changes permanent:


6.2. SRS (SENDER REWRITING SCHEME) 109

• Click on Update Global Settings

6.2 SRS (Sender Rewriting Scheme)

In order to avoid spurious SPF failures when CanIt-Domain-PRO forwards mail to a back-endserver that performs SPF checking, you can enable Sender Rewriting Scheme (see http://en.wikipedia.org/wiki/Sender_Rewriting_Scheme for a description of Sender RewritingScheme.)

To enable Sender Rewriting Scheme, you must perform the following steps:

• Pick a domain to use for the SRS addresses. This domain should not be currently in use foranything else. We recommend creating a subdomain of your existing domain solely for usewith SRS. For example, if you own the domain example.com, then srs.example.comwould be a good choice.

• Publish MX records for the SRS domain that point to the CanIt-Domain-PRO scanner or scan-ners.

• Under Administration, enter the SRS domain as the value of G-11000 SRS Domain

• If and only if you are not running a CanIt-Domain-PRO appliance, perform the following steps:

1. Update the Sendmail access map to permit relaying for the SRS domain.

2. Add a mailertable entry for the SRS domain and set the mailer to local:srshandler

3. Create a Sendmail alias directing srshandler to"|/usr/share/canit/scripts/canit-srs-bounce-handler"

If you are running a CanIt-Domain-PRO appliance, the above steps are done for you automati-cally.

Additionally, you must specifically enable SRS on a per-stream basis (following the usual CanIt-Domain-PRO inheritance rules.) To turn on SRS for a stream, enable setting S-930 “Enable SRS(Sender Rewriting Scheme)” under Preferences : Quarantine Settings.

Note: SRS requires Sendmail 8.14 or newer. If you are not running a CanIt-Domain-PRO appliance, makesure you have a new enough version of Sendmail.

Once SRS is enabled, CanIt-Domain-PRO will rewrite envelope senders that receive SPF “pass” toaddresses within the SRS domain. CanIt-Domain-PRO will also handle bounces to those addresses,restoring the original recipient.

Here are a few items to note about SRS:

• CanIt-Domain-PRO does not apply SRS to mail that was forced into a stream by a KnownNetworks entry. Such mail is typically outbound mail; in this case you should simply includethe outbound CanIt-Domain-PRO relays’ IP addresses in the domain’s SPF record.





• If you have back-end servers that forward inbound mail back out via CanIt-Domain-PRO (thiscan happen, for example, if some users on the back-end server configure their accounts to for-ward everything to Gmail or to Hotmail) then you should enable SRS on the inbound mail forthose users.

• CanIt-Domain-PRO applies SRS only if the original inbound mail received and SPF “pass”.

6.3 Real-Time DNS Blocklists


Both Sendmail and CanIt-Domain-PRO can make use of DNS-based real-time blocklists. Theseblocklists allow you to look up the IP address of a host in a special DNS domain, and take actionif the host is on the list.

You can configure Sendmail to use DNS-based blocklists directly, but you may prefer to handle thiswith CanIt-Domain-PRO, because CanIt-Domain-PRO allows you to hold or score messages fromhosts on the blocklist rather than outright rejecting them.

6.3.1 Entering the Master List of DNS RBLs

To use DNS-based RBLs, you first enter a master list of RBLs that CanIt-Domain-PRO can potentiallyuse. To do this, click on Administration and then Master RBLs. The Master RBLs page appears:

Figure 6.2: Master RBLs

To enter an RBL:

1. Enter the domain in the RBL Domain box.

2. Enter a brief (but meaningful) description in the Description box.

3. Enter a short tag in the Tag box. This tag is used in the mail log and incident reports to identifythe RBL. If you leave it blank, CanIt-Domain-PRO will construct a unique identifier for theRBL based on the domain, type and data.

4. Select how the RBL is to be used:


6.3. REAL-TIME DNS BLOCKLISTS 111

(a) A Block RBL is used to block unwanted mail. Users will be able to create “Ignore”,“Hold/Tag”, “Reject” or “Score” RBL rules. Any “Score” rule will have to have a non-negative score.

(b) An Allow RBL is used to list known good mail servers. Users will be able to create“Ignore” or “Score” rules, but any “Score” rule will have to have a non-positive score. Inaddition, no extra greylist delay may be created for an Allow RBL.

5. Select the type of addresses listed by the RBL:

(a) If you know that the RBL lists only IPv4 addresses, set the Address Family to IPv4.(b) If you know that the RBL lists only IPv6 addresses, set the Address Family to IPv6.(c) If the RBL lists both IPv4 and IPv6 addresses, set the Address Family to Both IPv4 and

IPv6. If you are not certain whether or not the RBL lists IPv6 addresses, the “Both” settingis safest.

6. Select the type of the RBL:

(a) If the RBL is considered to be “hit” if any record is returned, set the type to normal. MostDNS-based blocklists are of this type.

(b) If the RBL returns specific A records to indicate a hit, set the type to match and enter theA record that indicates a hit in the Data field. As a special case, you can use an X in placeof an octet to allow a wildcard match. For example, a data field of 127.0.X.3 wouldmatch an A record of 127.0.0.3, 127.0.1.3, 127.0.55.3, etc.

(c) If the RBL returns information in a bitmask in the returned A record, set the type tomask and enter the mask (for example, 0.0.0.4) in the Data field. A mask-type RBL isconsidered to be hit if the returned A record bitwise-ANDed with the data field returnsnon-zero.


To delete an RBL, enable the checkbox beside the entry you wish to delete and click Submit Changes.Deleting a master RBL also deletes all RBL rules that refer to it.

You can change the timeout for RBL lookups by adjusting the value in the Timeout in seconds forDNS-RBL lookups box.

The master RBL list is merely a list of all the RBLs that CanIt-Domain-PRO can potentially use. Toactually set up RBL rules, please see the User’s Guide. RBL rules can be created on a per-streambasis, so different streams can elect to use none, some or all of the predefined Master RBLs.

Note: Various RBLs have different terms-of-service. Some require licensing or payment; please be sure youare allowed to use an RBL before entering it into CanIt-Domain-PRO’s RBL list.

6.3.2 combined.bl.rptn.ca

Roaring Penguin Software Inc. publishes for DNS-based lists for CanIt-Domain-PRO customers.These lists are automatically entered into the Master RBL list (but no rules are created automatically.)

The four lists are:



• The Greylist-Stumbler list. These are machines known to have trouble getting past greylisting.The machines are very likely compromised PCs. We recommend making a rule to add 0.5points for machines on this list, and also to extend the greylist period (if you use greylisting) to60 minutes.

• The Dictionary-Attacker list. These are machines known to send mail to many nonexistentaddresses. We recommend a rule to add 0.5 points for machines on this list.

• The Spam-Source list. These are machines known to send spam and relatively little non-spam.We recommend adding three points for machines on this list.

• The Mixed list. These machines send both spam and non-spam; we recommend adding 1.5points for these machines.

Note: The combined.bl.rptn.ca list requires a secret token for lookups to succeed; this token is changedonce a day. CanIt-Domain-PRO automatically obtains and uses the token for as long as your supportterm is in force. This means that you cannot use the list outside of CanIt-Domain-PRO. If you do ahigh volume of lookups, please contact Roaring Penguin Software to arrange for a zone transfer viarsync.

6.4 Phishing URLs

Note: The ability for end-users to vote URLs as malicious is available only if you have enabled CanIt StorageManager (Chapter 16)

CanIt-Domain-PRO maintains a list of URLs that are known to be malicious or to have been used inphishing messages. There are two sources of these URLs:

• A large list is distributed by Roaring Penguin to each CanIt-Domain-PRO installation. YourRPTN credentials provide access to this list.

• Each CanIt-Domain-PRO administrator can additionally maintain a local list of phishing URLs.

6.4.1 Malicious URL Votes

When end-users reject an incident from the quarantine page, they can choose merely Reject messageor the stronger Reject and Report Phish/Fraud. The latter presents users with a list of URLs in therejected message and asks them to indicate which URLs they believe to be malicious. Each such URLis entered as a phishing URL vote.

You can review phishing URL votes by clicking on Administration and then Phishing URLs. ThePhishing URL Votes page appears:


6.4. PHISHING URLS 113

Figure 6.3: Phishing URL Votes

This page shows all of the URLs that users have indicated are malicious. The various columns in thedisplay are:

• URL — a normalized version of the URL with any leading http: or https: stripped. Notethat URLs longer than 40 characters are truncated and an ellipsis (...) is placed after them; hoverthe mouse pointer over the URL to see the full URL.

• Votes — the number of times the URL has been voted as malicious.

• In Phishing List? — Set to “No” if the URL is not in the central list of known phishing URLs,or “Bad” if it is. If the URL has query parameters (for example: example.com/foo?x=1)and the base URL example.com/foo without query parameters is in the central list, thenthis column will contain “Base URL Bad”.

If the URL is in the known phishing list, then the source is indicated as local or RPTN:*.local means the URL was added by the local CanIt-Domain-PRO administrator; RPTN:*means it came from the central Roaring Penguin list. The specific text after RPTN: providesadditional detail about the source of the URL.

• Last Vote — the date the URL was last voted as being malicious.

• Action — a list of actions to take against the URL. Possibilities are:

– Do Nothing — don’t take any action.

– List URL as Bad — enter the URL into the known phishing URL list, marked as mali-cious.

– List base URL as Bad — remove the query parameters, if any, from the URL and enter itas a malicious URL in the known phishing URL list.

– List URL as OK — explicitly indicate that the URL is not malicious. You can use this if,for some reason, you need to override a URL marked as malicious in the central RoaringPenguin list.

Note that any URL you enter into the list of known phishing URLs will be set to expire after120 days. You can alter this expiry date as described in Section 6.4.2.

• Delete? — this permits you to delete all votes relating to the URL. Note that if the URL is inthe known phishing URL list, deleting it from the Phishing URL Votes page does not remove itfrom the list. It merely deletes all users’ votes pertaining to the URL.



If you take action against phishing URLs or delete any, click Submit Changes to make your changestake effect.

Filtering the List of Phishing URL Votes

You can filter the list of phishing URL votes displayed as follows:

• Enter a string in the “Entry Contains:” filter box to restrict URLs to those containing a particularstring.

• Enter a positive integer in the “Minimum Votes:” filter box to restrict URLs to those with atleast that many votes.

• Select one of “Any”, “Yes” or “No” from the “In Known-Phishing List?” pulldown to restrictthe URL display to those which meet the filter condition.

Once you have created filter conditions, click Filter to apply them.

6.4.2 Known Phishing URLs

To see the entire list of URLs known or suspected to be malicious, click on Administration and thenPhishing URLs. In the Phishing URL Votes page, click on Known-Phishing List in the third-levelmenu. The Known Phishing URLs page appears:

Figure 6.4: Known Phishing URLs

The list of phishing URLs has eight columns:

• URL — a normalized form of the URL. Note that URLs longer than 40 characters are truncatedand an ellipsis (...) is placed after them; hover the mouse pointer over the URL to see the fullURL.

• Votes — the number of times a URL has been voted as malicious by a local user.

• Status — “Bad” if the URL is considered malicious; “Good” if it is considered harmless.

• Source — the source that determined the URL to be malicious. Possible values for Source are:


6.4. PHISHING URLS 115

– local — the URL was marked as malicious by the local CanIt-Domain-PRO site adminis-trator.

– RPTN:APER — the URL was considered malicious by the Anti-Phishing Email Replyproject at https://code.google.com/p/anti-phishing-email-reply/.

– RPTN:Phishtank — the URL was considered malicious by the Phishtank project athttp://www.phishtank.com/.

If Roaring Penguin adds additional feeds of malicious URLs, there may be additional values forSource, but all of them will start with RPTN:.

• Last Vote — the time of the most recent vote by a local user (if there was one) that the URLwas malicious.

• Expiry — the date when the URL will expire and be auto-deleted from the list. By default, localentries expire 120 days after they are created. RPTN entries do not expire, but are removed ifthe URL is removed from the central RPTN lists maintained by Roaring Penguin.

• Action — an action to take against the URL. Possible actions are Do Nothing, List URL asBad and List URL as OK, all of which are self-explanatory.

• Delete? — a checkbox for deleting a URL from the known phishing URL list. Note: If youdelete a URL with a source other than local, it will reappear next time CanIt-Domain-PROupdates its URL list from Roaring Penguin’s data feed.

If you make any changes (taking action against URLs, changing the expiry date or deleting any URLs),click Submit Changes to make them take effect.

If you wish to add URL that is not currently in the Known Phishing URLs list, you can enter it in thetop row in the URL column and hit Submit Changes to add it to the list manually.

Filtering the Known Phishing URL List

You can restrict which URLs are displayed by entering text into the “Entry contains:” and/or “SourceContains:” filter boxes and clicking Filter.

6.4.3 Delaying Messages because of local Phishing Votes

There can be a significant delay between the time a URL is voted on by end-users as fraudulent andthe time the administrator adds it to the Known Phishing list. To mitigate problems caused by thisdelay, CanIt-Domain-PRO allows you (on a per-stream basis) to delay messages once the URLs inthem have a certain number of phish votes.

Under Preferences : Quarantine Settings, set S-1630 to the minimum number of votes to trigger adelay. A value of 0 disables the feature. We recommend a value of at least 5 so that messages areunlikely to be delayed because of a couple of incorrect votes.

On that same page, set S-1640 to the number of hours to delay the message.


https://code.google.com/p/anti-phishing-email-reply/

http://www.phishtank.com/


If a message comes in containing a URL that has at least as many phish votes as the S-1630 setting,then it is put into a special stream called @@DELAYED in the recipient’s realm. After the numberof hours specified in S-1640, the message will automatically be released from @@DELAYED and res-canned.

We recommend setting a notification address in the @@DELAYED stream to notify administratorshourly. That way, they can check that stream’s quarantine and reject malicious messages before theyare released. Administrators can also take the opportunity to add malicious voted-on URLs to theKnown Phishing URL list.

6.5 Users

CanIt-Domain-PRO maintains its own table of users. You should enter users into this table to createCanIt-Domain-PRO administrative users, or users with different privileges from the default (for ex-ample, a demo user.) Click on Administration and then Users to set up users. You will see the usermanagement screen.

Figure 6.5: Users

If you enter a string in the “Filter:” box, then CanIt-Domain-PRO limits the display to entries whoseUser-ID or E-Mail column contain that string.

We recommend using the form [email protected] for user names. Users whose names follow thatformat will be placed in the appropriate realm, as determined by looking up domain.com in theRealm Mapping Table. A user name of the form user without a domain will normally be placedin the base realm. (However, a user can specify a particular realm to log in to by logging in asrealm:user.)


6.5. USERS 117

6.5.1 User Privileges

When a user logs in to CanIt-Domain-PRO, he or she can see a single stream at a time. Everyuser always has access to a stream that (usually) has the same name as his user name. The CanIt-Domain-PRO administrator can give users permission to see additional streams. For example, the userjanedoe always has access to the stream janedoe. However, if she manages a mailing list calledjoke-list, you have two options:

1. You can stream messages for the list to janedoe, so she has only a single spam quarantine toconsider.

2. You can create a new stream called joke-list and give access to that stream to janedoe.In this way, she can use different settings, blocklists and allow-lists for the list than she does forher personal e-mail.

Each CanIt-Domain-PRO user has two special privileges, which can be on or off:

• A user with root privilege can add, edit and delete other users. A user with root privilege inthe base realm has overall System Administrator privileges. A user with root privilegein any other realm has Realm Administrator privileges. The overall System Administrator cansee (and create) users in other realms. Realm Administrators can only create users in their ownrealms.

• A user with write privilege can mark messages as spam or not-spam, and can block and allowhosts, domains and senders. A user without write privilege is called a read-only user and cannotmake any changes whatsoever. A read-only user can look, but not touch.

Note that CanIt-Domain-PRO allows for additional flexibility in controlling which parts of the Webinterface are available to various users. For details, see Chapter 9.

6.5.2 Adding a User

To add a user, click on the Add User link. The Add User screen appears:



Figure 6.6: Add User

• Enter the user-ID of the user in the User-ID box.

• Select the realm for the new user from the Realm pull-down menu or enter it into the realm box.If you leave the realm blank, then a realm will automatically be assigned based on the user-ID.If the user-ID looks like an e-mail address, the realm is chosen by mapping the domain-namepart of the user-ID to a realm.

• To set the user’s e-mail address, enter it in the E-Mail field. (If CanIt-Domain-PRO knows auser’s e-mail address, the “Locked Addresses” feature can be used.)

• Enter a password for the user in the Password and Confirm Password fields.

• If you set Locked Password? to Yes, then the user will have a “locked” password and will notbe able to log in. However, if you have configured an alternate user authentication method, theuser will be able to log in using a password that the alternate method accepts.

• If you only want the user to have read-access to the spam quarantine, set Write Access? to No.

• If you want to make the user an administrator in his realm, set Has Root Access? to Yes.

Once you have filled in the fields, click Add User to add the user.

Note: Both user-names and passwords are case-sensitive; a used named user1 is completely different fromone named User1.

6.5.3 Editing a User

To edit a user, click on the User-ID on the user management screen. You will see the user-editingscreen.


6.5. USERS 119

Figure 6.7: Edit User

• The user’s realm may be displayed, but it cannot be edited once the user has been created.

• To set the user’s e-mail address, enter it in the E-Mail field.

• If you wish to change the user’s password, enter it in the Password and Confirm Passwordfields. If you leave these fields blank, the password will not be changed.

• If you set Locked Password? to Yes, then the user will have a “locked” password and will notbe able to log in. However, if you have configured an alternate user authentication method, theuser will be able to log in using a password that the alternate method accepts.

• Adjust the write-access privilege by setting the Write-Access? checkbox appropriately. (If youare editing the currently logged-in user, you can’t change the Write-Access setting.)

To make the changes take effect, click Submit Changes.

6.5.4 Deleting a User

If there is more than one user, a Delete checkbox appears beside those users that can be deleted.Enable the checkbox and then click Submit Changes to delete the selected user or users. Note that itis not possible to undo the deletion!

Note that if you delete a user, he may still have access if he can be authenticated using an externalauthentication mechanism.

6.5.5 Granting Access to Streams

If you wish to grant a user access to additional streams, click on the Edit Accessible Streams button(Figure 6.7). The following page will appear:



Figure 6.8: Granting Access to Streams

To grant access to a stream, enter the stream name in the input box and click Add Stream. To revokeaccess to a stream, enable the Delete checkbox next to the stream name and click Delete SelectedStreams. If you grant access to a stream named * (a single asterisk), then the user is given access toall streams in his or her realm.

Note that a user always has access to a stream with the same name as his user name, and this accesscannot be revoked. Also, the CanIt-Domain-PRO administrator can access any stream, regardless ofthe settings on this page.

6.5.6 Switching Users

CanIt-Domain-PRO permits an administrative user to switch to another user-ID. This is useful if youwant to see the interface exactly as another user would see it. A realm administrator can switch to anyuser within his own realm or any realm in the subtree under that realm.

To switch users:

1. Click Administration : Switch User

2. Enter the user you wish to become in the User-ID box.

3. Enter the stream in which the user should be placed after the switch in the Stream box. Notethat CanIt-Domain-PRO does not run the normal user-lookup to determine a user’s home streamwhen you switch users; hence, you may need to enter the home stream explicitly.

4. If you own subrealms, you will be asked for the realm of the new user. Note that CanIt-Domain-PRO does not attempt to deduce the realm based on the User-ID; you need to explicitly select arealm in the Realm field.

5. Click Submit Changes. You are now logged in as the new user.

Note: Once you switch users, there is no going back. In most cases, you have to log out and log back inagain to become the original user. Also, if you are logged in as a read-only user, then you remainread-only no matter which user you switch to.


6.6. PERMITTING USERS TO OPT IN 121

6.6 Permitting Users to Opt In

In the CanIt-Domain-PRO global settings (Section 6.1), the CanIt-Domain-PRO administrator cancontrol:

• Whether or not people are permitted to opt-in to spam scanning.

• Whether the default setting is opt-in or opt-out.

There are three useful combinations:

1. Permit everyone to opt-in, and have the default be opt-in.

2. Permit everyone to opt-in, and have the default be opt-out.

3. Permit only selected people to opt-in, and have the default be opt-out.

In the first two cases, the administrator need not do anything special. In the third case, you must addentries to the Stream Approval Table. Click on Administration and then Opt Others In/Out to seethis table:

Figure 6.9: Stream Opt-In Approval

If the “Approved?” column is checked, then the stream may opt in to spam scanning. If it is notchecked, then the stream may not opt in to spam scanning.

If the “Opted-In?” column is checked, the stream is currently opted in to spam scanning. Otherwise,it is not.

To add a stream to the table, enter the stream name in the input box and set “Approved?” and “Opted-In?” appropriately. Then click Submit Changes.

To edit existing streams, adjust “Approved?” and “Opted-In?” appropriately and click SubmitChanges. To delete a stream from the opt-in table, enable the Delete? checkbox on the appropri-ate row and click Submit Changes.



If the default setting is to permit anyone to opt in to spam scanning, you can nevertheless excludeparticular streams from being able to opt in by entering them in the Stream Approval Table and turningoff the “Approved?” checkbox.

In order for spam-scanning to occur, a stream must be both approved and opted-in. If the stream is notfound in the Stream Approval Table, then the defaults are taken from the Global Settings.

If you enter a string in the “Filter:” box, then CanIt-Domain-PRO limits the display to entries whoseStream column contains that string.

6.7 Groups

For the purpose of granting permissions, CanIt-Domain-PRO allows you to create groups. A group issimply a collection of users.

To edit groups, click on Administration and then Groups. The Groups Page appears:

Figure 6.10: Groups

6.7.1 Creating, Deleting and Editing Groups

To create a new group:

1. Enter the name of the group in the Group box.

2. Enter a description of the group in the Description box.


To delete an existing group:

1. Enable the Delete checkbox for the group you want to delete.


To edit a group:


6.8. VIEWING ACTIVE STREAMS 123

1. Click on the Edit link next to the appropriate group. The Group Members page appears:

Figure 6.11: Group Members

2. Enter new members (one per line) in the Member text area.

3. If you want to delete existing members, enable the appropriate Delete checkbox.


Note: External authentication methods can affect group membership. See Chapter 7 for details.

In the Groups Page (Figure 6.10), click on Permissions to edit the permissions associated with thegroup. Permissions will be discussed in detail in Chapter 9.

6.8 Viewing Active Streams

The CanIt-Domain-PRO administrator can look at all the streams with entries in the incidents table.To do this, select Administration and then See Active Streams. The Active Streams Page appears:



Figure 6.12: Active Streams

6.8.1 Definition of an Active Stream

A stream is considered “active” if it has at least one message in the quarantine (pending, spam ornon-spam) or has any rules, blocks or allow rules defined.

6.8.2 The Active Stream Display

The columns in the display are:

Stream The name of the stream. Each stream name is a hyperlink; if you click on the link, you willswitch streams to that stream.

Pending The number of pending messages in the stream’s quarantine.

Spam The number of spam messages in the stream’s quarantine.

Non-Spam The number of non-spam messages in the stream’s quarantine.

Opted-In? Set to Yes if the stream is both approved for anti-spam scanning and opted-in; set to Nootherwise.

Delete A column of links for deleting streams.

If you enter a string in the “Filter:” box, then CanIt-Domain-PRO limits the display to entries whoseStream column contains that string.


6.9. FILTERING OUTBOUND MAIL 125

6.8.3 Deleting a Stream

To delete a stream, click on the Delete link in the Active Streams page. Then click on Yes, delete it!to confirm deletion. Deleting a stream deletes all incidents, rules, settings, etc. associated with thestream.

6.9 Filtering Outbound Mail

Some organizations like to add boilerplate disclaimers to outbound mail. CanIt-Domain-PRO canachieve this by streaming all outbound mail to an “outbound” stream, and adding boilerplate optionsfor that stream.

One way to stream all outbound mail to a particular stream is to set up your domain mappings asfollows:

• All of your own domains (that is, domains considered “internal”) should have mappings set up.The mappings could be ChopDomain, Sendmail, or whatever, as long as the mappings exist.

• The wild-card domain * should have a domain mapping of Database.

• The wild-card address * should have an address mapping mapping it to the stream outbound.(You can name your outbound stream however you like.)

With these settings, mail for internal recipients will be streamed appropriately, and mail for externalrecipients will all be streamed to outbound.

For the outbound stream, enter the appropriate boilerplate to add to outbound messages. You can alsoadd custom body-matching rules if you want to quarantine mail containing certain words—for exam-ple, “Do Not Distribute Externally” Such rules on an outbound stream may help prevent unauthorizeddistribution of confidential information.

See also Known Networks (Section 5.7 on page 65) for another way to force outbound mail into aspecific stream. Using Known Networks may be simpler than using address mappings if all youroutbound mail originates from a limited set of IP addresses.

6.9.1 DKIM-Signing Outbound Mail

DKIM, or DomainKeys Identified Mail is a mechanism for proving that a particular organization’sservers relayed a mail message. More specifically, DKIM uses cryptographic techniques that allowrecipients to validate that a specific domain is indeed associated with the message.

CanIt-Domain-PRO permits you to DKIM-sign outbound mail. Please note that the only way CanIt-Domain-PRO can validate the origin of a message is to look at the sending IP address. Therefore,CanIt-Domain-PRO DKIM-signs messages based on a domain being associated with a Known Net-works entry with the “Force-to-Stream” parameter (indicating outbound mail) set.

For example, consider this Known Networks entry:


http://dkim.org/


Figure 6.13: Known Network with Associated Domains

In this example, CanIt-Domain-PRO will permit DKIM signing of a message:

• That has a From: header sender in the domain example.com or example2.net

• Providing also that the message originates from 192.168.7.88/32

In order to DKIM-sign a message, CanIt-Domain-PRO requires a key pair to be generated. A key pairconsists of two cryptographic keys that work together: The private key is a very large number that iskept secret. It is used to sign a message. The public key is another very large number that is connectedto the private key and must be made public; anyone who has the public key can verify whether or nota message was indeed signed by the private key. The mathematics of signing is such that althoughpossession of the public key permits verification that a message is signed, only possession of theprivate key permits the actual signing process. Additionally, it is believed that it is infeasible to derivethe private key given only the public key.

A DKIM public key is typically published by creating a special TXT DNS record that contains thepublic key and a few ancillary pieces of information required by DKIM verification software.

Managing DKIM Keys

To DKIM-sign a message from a particular domain, CanIt-Domain-PRO needs a key pair. To generatea key pair, click on Setup and then DKIM Keys. The DKIM Key List screen appears:

Figure 6.14: DKIM Key list


6.9. FILTERING OUTBOUND MAIL 127

In Figure 6.14:

• The domain example.com has a single DKIM key pair. The DKIM selector is canit andthe key is active, meaning it will be used to sign outbound mail.

• The domain example.net has a single DKIM key pair whose selector is sel1. That key isnot active, so it will not be used to sign outbound mail. Since example.net has no activekeys, its outbound mail will not be signed at all.

• The domain example.org has two DKIM key pairs: One with selector canit and anotherwith selector canit2. The canit key is active, so it will be used for signing.

To add a new DKIM key pair, click on Add New DKIM Key Pair. The DKIM Key Pair page appears:

Figure 6.15: Adding a DKIM Key Pair

Enter a domain name and click Save. Note that the domain you enter must be associated with at leaston Known Networks entry. Additionally, the domain must be within the current realm.

The DKIM selector defaults to “canit”, but you can use any selector you like as long as it is at most16 characters long and can appear as a legal domain name component.

Once you have added the key pair, CanIt-Domain-PRO will display information about the key:

Figure 6.16: DKIM Key Details

The information displayed includes the domain name DNS TXT record required to publish the publickey. Please note: DKIM TXT keys are typically quite long. We display them in BIND 9 multi-part



string format. Different DNS software might require the record to be entered in a different format; inreality, the entire record is one long piece of text. Please consult your DNS provider’s documentationfor more information on the format required for DKIM keys.

Activating a DKIM Key Pair

When you first create a DKIM key pair, it is not active. To activate the key pair, return to the DKIMkey list and enable the Active radio button. Then click Submit Changes. If the domain has any otherDKIM key pairs, they are automatically deactivated.

Deleting a DKIM Key Pair

To delete DKIM key pairs, click on Setup and then DKIM Keys. Enable the appropriate checkboxesin the Delete? column and click Submit Changes.

DKIM Selectors

CanIt-Domain-PRO allows you to specify a DKIM selector, but note that any given domain is onlyallowed to have one active DKIM key. DKIM selectors are useful should you wish to roll over yourkeys. Here is an example:

• Suppose you create a DKIM key pair with the selector s201501. CanIt-Domain-PRO createsthe key pair and you publish a DNS record.

• Sometime later, you want to change the key pair because it’s good practice to change keysevery now and then. Within CanIt-Domain-PRO, create a new key pair with a selector of (forexample) s201506. Leave the old record for 201501 in place on your DNS server and publishan additional DNS record for the new s201506 key pair.

To roll over to the new key, simply make it active; the old key will automatically be deactivated.

In this way, old messages can still be verified for as long as you keep the s201501 DNS recordin place, but all new messages will be signed and verified with the new key pair and the selectors201506.

Selectively DKIM-Signing Outbound Mail

Sometimes, an organization may not wish to DKIM-sign all of its outbound mail. CanIt-Domain-PROlets you selectively sign outbound mail as follows:

When CanIt-Domain-PRO sees an outbound message, it computes the stream that the From: headeraddress would be in. If, for some reason, it cannot find the stream, it uses the default stream in therealm of the From: header address.

If the Quarantine Setting S-1050 Enable DKIM Signing for outbound messages originating fromsenders in this stream is set to Yes in the stream determined above, CanIt-Domain-PRO DKIM-signs


6.10. COPYING RULES FROM ONE STREAM TO ANOTHER 129

the message. Otherwise, it does not. This allows you to avoid DKIM-signing bulk messages, auto-mated messages, etc. providing they originate from addresses with their own streams.

By default, S-1050 is set to Yes, so by default outbound mail is DKIM-signed if a key pair is presentand the messages comes from an associated domain of a Known Network.

A Note on some DKIM-Signing Pitfalls

CanIt-Domain-PRO uses software called Sendmail to actually accept and deliver messages. Send-mail may make its own header modifications to messages without CanIt-Domain-PRO’s knowledge,thereby breaking DKIM signatures. In most cases, programs used to compose email messages doso in such a way that Sendmail does not need to modify anything and DKIM works fine. But werecommend testing DKIM with all the mail software your users employ to ensure it generates correctsignatures.

In particular, Sendmail will change a header that looks like this:

From: Full Name <[email protected]>

to this:

From: "Full Name" <[email protected]>

Note the additional double-quotes. Most email software generates the header with double-quotes inplace anyway, so Sendmail has no need to modify the header. If your software does not, you’ll need away to force it to do that if you wish to have CanIt-Domain-PRO DKIM-sign your messages.

6.10 Copying Rules from One Stream to Another

Occasionally, it is useful to copy or move rules from one stream to another. To do this, click onAdministration and then Copy Rules. The Copy Rules page appears:

Figure 6.17: Copying Rules

To copy rules:

1. Choose which rules you wish to copy by activating the appropriate check boxes under Objects



to Migrate.

2. Put the name of the stream you want to copy from in the From stream: box.

3. Put the name of the stream you want to copy to in the To stream: box.

4. Select “Preserve Original” or “Overwrite” to handle the case of conflicting rules in the sourceand destination streams.

5. Click on Copy Rules to copy rules from the source stream to the destination stream. MoveRules is similar, but any rule that is successfully placed in the destination stream is deletedfrom the source stream.

6.11 Secondary MX Hosts

Secondary MX hosts require special handling by CanIt-Domain-PRO. Secondary MX hosts which re-lay to the CanIt-Domain-PRO system should always be listed in “Known Networks”, with the optionsbelow checked, as it is usually desirable to modify CanIt-Domain-PRO behaviour as follows:

Note that localhost (127.0.0.1) is always considered a secondary MX host for the purposes below:

Friendly Host When checked, rejected mail is simply discarded rather than being failed with a 5xxcode. This prevents the friendly host from generating backscatter.

Parse Received Headers When checked, CanIt-Domain-PRO trusts the Received: header added bythat connecting host or network. This means that CanIt-Domain-PRO will be able to apply hostchecks against the host that submitted the message to your network, rather than against yoursecondary MX server.

Prohibit Block Rules When checked, CanIt-Domain-PRO ignores any host blocks for hosts in thisnetwork. This will prevent locally-generated mail from your secondary MX hosts from beingblocked. Note that if “Parse Received Headers” is enabled, mail relayed via the secondarysystem will show as being from the upstream IP, and blocks will not be ignored.

Skip RBL Lookups When checked, CanIt-Domain-PRO will suppress DNS blocklist lookups.

Skip Greylisting When checked, CanIt-Domain-PRO will suppress first-time sender checks.

Any machine under your control that you expect to forward mail to your machine should be considereda secondary MX host. For example, if a number of users have accounts on a machine that forward mailto your machine using .forward files, you should consider entering that machine as a secondary MXhost.

Also, note that if CanIt-Domain-PRO is able to determine the “real” relay IP by parsing the Received:headers, and you have enabled this option, then CanIt-Domain-PRO runs all the host checks as usual,using the real relay IP address. However, these checks are (of necessity) delayed until after the DATAphase of the SMTP transaction, because CanIt-Domain-PRO does not have the required informationat the MAIL FROM: or RCPT TO: phases.


6.12. AVOIDING BACKSCATTER 131

6.12 Avoiding Backscatter

Under most circumstances, if CanIt-Domain-PRO rejects a message, it responds with an SMTP failurecode. This generally causes the sending relay to mail a failure notification to the original sender.

However, because most spam and viruses have faked sender addresses, you may not want this behaviorfor messages relayed from a secondary MX host or for messages split into multiple streams. That’sbecause if a message is rejected after having been accepted by one of your mail servers, it’s theresponsibility of the sending server to generate a failure Delivery Status Notification or DSN.

If (as is likely) the sender address is faked, that failure message may arrive at an unsuspecting third-party. This is what is known as backscatter.

It is a violation of RFC 821, and is generally considered bad behavior, to silently discard mail; how-ever, many sites are beginning to lump hosts responsible for generating backscatter into the samecategory as spammers. Because of this, CanIt-Domain-PRO will not generate a failure notification formail from local host or from a designated secondary MX host.

6.13 Test Plugins

Some anti-spam tests are very specific and are implemented as plugins. Currently, CanIt-Domain-PROships with a number of plugins that are described in subsequent sections.

If a plugin matches against a particular message, the plugin is said to have fired.

To configure test plugins, click Rules and then Plugins. The Test Plugins page appears:

Figure 6.18: Test Plugins

For each plugin, you can configure actions to be taken on a per-stream basis (although we recommendcreating rules only in the default stream.)

To configure a plugin:

1. Select the action to be taken if the plugin fires. The action can be one of:



• Ignore — do not use this plugin at all.• Hold/Tag — hold mail in the quarantine if the plugin fires. (In a tag-only stream, this will

be converted to a tag.)• Score — add the score in the Score column if the plugin fires.• Reject — reject the mail if the plugin fires.

2. If you chose Score, enter a decimal score in the Score column.

3. If you wish, you can add a comment in the Comment column

4. Click Submit Changes to make the changes take effect.

6.13.1 The PhishingAddress Plugin

The PhishingAddress plugin consults a dynamically-updated list of e-mail addresses known to be usedin phishing attacks. We recommend configuring it as follows:

• In the default stream, configure the test to add 10 points to the message score. Alternatively,you may wish to configure it to reject mail.

• If you are routing outbound mail through CanIt-Domain-PRO, then you should be sending out-bound mail through a dedicated outbound stream. In that stream, configure the PhishingAddressplugin to reject mail. If users accidentally reply to a phishing e-mail that somehow got through,at least by rejecting their replies you will prevent sensitive information from reaching the at-tackers.

6.13.2 The PhishingURL Plugin

The PhishingURL plugin consults a dynamically-updated list of URLs known to be used in phishingattacks. It fires if a message contains one or more URLs on the list.

We recommend configuring the test to add 10 points to the message score. Alternatively, you maywish to configure it to reject mail.

6.13.3 The OfficeMacros Plugin

The OfficeMacros plugin examines Microsoft Office attachments and fires if they contain macros.Since office documents containing macros can be extremely dangerous and can be used to spreadmalware and ransomware, we recommend scoring this plugin at 3.5. However, if you find there aretoo many false-positives, cautiously lower the score.

6.13.4 The OfficeMacro* Open Plugins

Three plugins named OfficeMacroAuto Open, OfficeMacroDocument Open and OfficeMacroWork-book Open fire if a Microsoft Office document contains a macro with the name Auto Open, Docu-ment Open or Workbook Open, respectively. In addition, the Auto Open plugin fires if the MicrosoftOffice document appears to invoke powershell.exe, cmd.exe or shell.exe.


6.14. EMERGENCY BLOCKING OF DELIVERY STATUS NOTIFICATIONS 133

These macros are often used by malicious software to launch a virus payload. The default and recom-mended action for each plugin is to score 10 points.

Note that legitimate spreadsheets fairly frequently contain the Workbook Open macro, so you maycautiously reduce the score for OfficeMacroWorkbook Open, although we recommend doing it on acase-by-case basis (rather than in the default stream) to limit the risk.

6.13.5 The Shortener404 Plugin

The Shortener404 plugin fires if an email contains a URL on a known URL shortener such as t.co,bit.ly, tinyurl.com, etc. that returns a 404 “Page Not Found” response code.

This plugin can only score, not reject. Additionally, the score changes the behavior of the plugin:

• A negative score causes the plugin to be completely ignored; it is not run at all.

• A zero score causes the plugin to run. Any expanded URLs returned by the URL shorteningservices are added to the list of URLs to check against the Known Phishing URL database.

• A positive score causes the plugin to run; additionally, if any shortened URL returns a 404response code, then the score is added to the incident’s score.

6.13.6 The NewlySeenDomain Plugin

The NewlySeenDomain plugin fires if the envelope sender of a message is from a domain that CanIt-Domain-PRO first saw less than 7 days ago. This plugin is designed to treat new domains with somedegree of skepticism; we recommend scoring 1 point for newly-seen domains.

6.14 Emergency Blocking of Delivery Status Notifications

Sometimes, a spammer will process a large spam run and fake the sender address to be within adomain you control. Faking the sender address as if it comes from an innocent third-party is called ajoe-job.

Unfortunately, in a typical spam run, a large percentage of the recipient addresses are invalid, so therun creates many delivery failure notifications (officially called Delivery Status Notifications or DSNs).Because of the faked sender address, all of these notifications come back to you, the innocent third-party. These spurious failure notifications are called backscatter and can cause a huge load on yourCanIt-Domain-PRO scanners, as well as a huge annoyance for end-users.

CanIt-Domain-PRO has a feature that allows you to block DSNs for selected domains for a limitedtime. This is an emergency measure and should only be used for a limited time in the face of largeamounts of backscatter.

Normally, this feature is disabled. To enable the feature, click on Setup : Features and enable thePermit Emergency Blocking of Delivery Status Notifications feature.

Next, click on Rules : Block DSNs. The Block Delivery Status Notifications page appears:



Figure 6.19: Block Delivery Status Notifications Page

To turn on DSN-blocking for a domain:

1. Enter the domain name in the Domain box.

2. Pick an expiry date. The default expiry date is 5 days in the future. CanIt-Domain-PRO will notlet you pick an expiry date more than 30 days in the future.

3. If you wish, add a comment explaining why you are enabling DSN-blocking.


To edit the expiry date and comment for existing entries, change the text in the appropriate boxes andclick Submit ChangesTo remove DSN-blocking from a domain, enable the appropriate Delete? checkbox and click Sub-mit Changes.

Note: DSN blocking applies to all streams in the realm. In this respect, it is different from other entries inthe Rules menu which apply to a particular stream.

6.15 Removing All Rules and Settings from a Stream

On occasion, it may be necessary to delete all rules, blocks, allow rules, settings, etc. from a stream.If a novice user has created many such rules and settings, the stream may be unusable and a “factoryreset” advised. To delete all rules and settings from a stream:

1. Switch to the stream in question with the View This Stream button.

2. Click on Rules.

3. Click on the link after the phrase “To delete all rules and stream settings for stream streamname,click here.”

4. Click Purge Rules to delete all the rules and settings, or Cancel to cancel.

Note: It is not possible to purge all rules from the default stream.


6.16. PROVISIONING INFORMATION 135

6.16 Provisioning Information

CanIt-Domain-PRO keeps track of all the addresses and streams that have received mail in the last 30days. It can display this information so you can track the usage of the system.

To view provisioning information, click on Administration and then Provisioning. The Provisioningpage appears:

Figure 6.20: Provisioning Information

The rather large provisioning table contains a number of columns. The columns are as follows:

• Realm - the name of the realm. The realm tree starting at the current realm is displayed alongwith little arrows to indicate the hierarchical structure. Realm names are links which, if clicked,display provisioning rooted at that realm.

• Domains - a list of domains mapped to the realm. Each domain name is followed by a greencheckmark and the green letters “MX” if its MX records point to CanIt-Domain-PRO. If theMX records do not point to CanIt-Domain-PRO, then the domain name is followed by a red Xand the red letters “MX”. Note that MX records are checked once a night by the nightly cronjob, so the information displayed here may be slightly out of date.

If a domain does not correctly validate recipients, the MX indicator is followed by a yellowhazard sign warning of the problem. Note that provisioning information for non-validatingdomains will not be accurate.

• Expiry - the expiry date (if any) associated with the realm.

• Addresses This Realm Only - the number of addresses in the realm that have received emailin the last 30 days.

• Addresses Including Subrealms - the number of addresses in the realm and all of its descen-dants that have received email in the last 30 days.

• Streams This Realm Only - the number of addresses in the realm that have received email inthe last 30 days.

• Streams Including Subrealms - the number of addresses in the realm and all of its descendantsthat have received email in the last 30 days.



• Outbound Addresses This Realm Only - the number of outbound addresses in the realm. Arealm is considered to be using outbound filtering if any of its domains is associated with aKnown Networks entry (Section 5.7.) In this case, all of its inbound addresses are counted inthe outbound column. Otherwise, no addresses are counted in the outbound column.

• Outbound Addresses Including Subrealms - the number of outbound addresses in the realmand all of its descendants.

• Outbound Streams This Realm Only - the number of outbound streams in the realm. Thecriteria and counting rules for outbound streams are similar to those for outbound addresses.

• Outbound Streams Including Subrealms - the number of outbound streams in the realm andall of its descendants.

If the Archiving add-on is installed, the following columns are present:

• Archiving Streams This Realm Only - the number of streams in the realm that have archivingenabled.

• Archiving Streams Including Subrealms - the number of streams in the realm and its descen-dants that have archiving enabled. This item is formatted as a list of count/retention pairs. Forexample, if a realm and its descendants have 45 streams archiving for 12 months, 201 archivingfor 24 months and 16 archiving for 36 months, then the output will be:

45/12, 201/24, 16/36

• Archive Retention Months - the number of months for which archived mail is retained. Thiscan be set on a realm-by-realm basis.

If the Secure Messaging add-on is installed, the following columns are present:

• Secure Messaging Streams This Realm Only - the number of streams in the realm that havesecure messaging enabled.

• Secure Messaging Streams Including Subrealms - the number of streams in the realm and itsdescendants that have secure messaging enabled.

6.16.1 Computer-Readable Provisioning Information

To download the provisioning data in CSV format (suitable for importing into a spreadsheet), clickthe Download in CSV Format link at the bottom of the page.

To download the data as JSON (suitable for processing by many scripting languages), click the Down-load in JSON Format link at the bottom of the page.


Chapter 7

External Authentication

7.1 Introduction

In addition to its built-in user list, CanIt-Domain-PRO can authenticate users using external mecha-nisms. To enable the use of external authentication mechanisms, these basic steps must be followed:

1. A User Lookup must be defined. A User Lookup describes to CanIt-Domain-PRO how to lookup user information from an external source.

2. An Authentication Mapping must be created. An Authentication Mapping tells CanIt-Domain-PRO which User Lookup to use for a given domain. You can use different authentication mech-anisms for different domains, which gives CanIt-Domain-PRO considerable flexibility.

Some User Lookups can also perform streaming. That is, given an email address, they can returnthe name of the stream associated with that email address. The LDAP (Section 7.2.2) and Program(Section 7.2.4) User Lookups can perform streaming. Using a User Lookup to perform streaming isvery powerful; for example, you could use an LDAP lookup to stream all of a user’s aliases into hissingle stream.

CanIt-Domain-PRO also supports integration with Microsoft’s Azure Active Directory.

7.2 User Lookups

To create a User Lookup:

• Click on Setup and then User Lookups. You will see the User Lookup list:


138 CHAPTER 7. EXTERNAL AUTHENTICATION

Figure 7.1: User Lookup List

• Click on Add a New User Lookup, and the User Lookup Wizard appears:

Figure 7.2: User Lookup Wizard

• Pick a name for the User Lookup, and click Next. The User Lookup method selection screenappears:

Figure 7.3: User Lookup: Method Selection

• Enter a comment for the lookup method. The comment can be anything you like; its purpose isto document the method so you remember what it does.


7.2. USER LOOKUPS 139

• Select a lookup method. CanIt-Domain-PRO supports the following methods:

– POP3: CanIt-Domain-PRO authenticates users against a POP3 server.

– IMAP: CanIt-Domain-PRO authenticates users against an IMAP server.

– LDAP: CanIt-Domain-PRO authenticates users against an LDAP server. If you are creat-ing a new user lookup, then the LDAP choice is broken into four possibilities. The firsttwo are appropriate if you are authenticating against Active Directory and the last two areappropriate if you are authenticating against a generic UNIX LDAP server:

1. LDAP (Active Directory: Log in using Windows username @ domain): This choicepre-fills settings that are suitable for logging in using your Windows user-name orWindows user-name followed by @ and the domain name.

2. LDAP (Active Directory: Log in using email address): This choice pre-fills settingsthat are suitable for logging in with your email address.

3. LDAP (Generic: Log in using username @ domain): This choice pre-fills settings thatare suitable for logging in with your user-id (or user-id followed by @ and domainname.)

4. LDAP (Generic: Log in using email address): This choice pre-fills settings that aresuitable for logging in with your email address.

Note: Once an LDAP user lookup is created, editing it shows the method as simply LDAP. Thefour possibilities enumerated above are simply conveniences that pre-select appropriatesettings when you first create the user lookup.

– Azure Active Directory integrates with Microsoft’s cloud-based Azure Active Directory.Note that you can use Azure Active Directory for streaming only, and not for authentica-tion.

– Program: CanIt-Domain-PRO invokes a program (that you supply) to perform authenti-cation.

– Program (Legacy method): CanIt-Domain-PRO invokes external programs in the sameway as older versions did (using the “Alternate Authentication” global setting that hassince been removed.)

– Rewrite: This method cannot be used for authentication; it can only be used for streammapping. It generates a stream name using simple rewriting rules on the email address.

• Normally, User Lookups may only be used by domains within the realm in which the UserLookup is defined. However, if you set “Allow subrealms to use this User Lookup?” to Yes,then domains in subrealms will be able to use the User Lookup. This is useful, for example,if you have a number of customer realms that are all back-ended on the same LDAP or IMAPserver.

• Click Next.

7.2.1 IMAP and POP3 Authentication

If you selected IMAP or POP3 authentication methods, then the wizard looks like this:



Figure 7.4: IMAP/POP3 User Lookup

To complete the setup:

• Enter the IP address or fully-qualified host name of the IMAP or POP3 server. If the server islistening on a non-standard port, add a slash followed by the port number to the server name.For example, if you have an IMAP server listening on port 1143 on the host magnesium, youcould enter magnesium/1143 as the server.

• If you would like to strip the domain name from the login name before attempting authen-tication, set the “Strip domain name” setting to Yes. If someone logs in to CanIt-Domain-PRO as [email protected] and this setting is Yes, then the username passed to the IMAPor POP3 server is simply user. (The default home stream, however, is normally the [email protected].)

• If you would like to strip the domain name from the home stream, set “Strip domain namefrom home stream after authentication?” to Yes. This means that if someone logs in [email protected], her home stream will be user.

• If you would like CanIt-Domain-PRO to force user-names authenticated by POP3 or IMAP tolower-case, set “Force user name to lower-case” to Yes. (This also implicitly sets the homestream name on login to lower-case.) The user name is lower-cased before being presented tothe POP3 or IMAP server.

• If you would like CanIt-Domain-PRO to force stream names (as determined by the POP3 orIMAP username) to lower-case, set “Force stream name to lower-case?” to Yes. If you want topreserve mixed-case stream names, set this setting to No (which is the default.)

• If you want CanIt-Domain-PRO to validate the SSL certificate of the server (assuming SSL orTLS is used), set “Validate server certificate” to Yes.

• Pick the appropriate encryption settings for CanIt-Domain-PRO to use when communicatingwith the POP3 or IMAP server.



• By default, when a user successfully logs in via POP3 or IMAP, CanIt-Domain-PRO caches theusername and encrypted password for 5 days. If your POP3 or IMAP server ever goes down,this permits users to continue to log in to CanIt-Domain-PRO (provided they have logged insuccessfully within the past 5 days.) You can change the cache time by editing “Number ofdays to cache successful credentials”. If you set this parameter to zero, then CanIt-Domain-PRO will not cache credentials upon successful login.

• By default, a user logging in as [email protected] is put into the [email protected]. If you wish to rewrite the stream using a more sophisticated mech-anism than simply stripping the domain, enter a rewrite expression for “Rewrite expression totransform login name to stream name:” (Rewrite expressions are described in Section 7.2.7.)

For example, suppose example.org and example.net are aliases. You want users to login as either [email protected] or [email protected], but always want the stream tobe [email protected]. In this case, use a Rewrite Expression of %[email protected].

• Click Next to see a summary of your settings.

• If all of the settings are correct, click Finish to create the POP3 or IMAP User Lookup.

7.2.2 LDAP Authentication and Streaming

If you are creating a new user lookup, then the LDAP choice is broken into four possibilities. The firsttwo are appropriate if you are authenticating against Active Directory and the last two are appropriateif you are authenticating against a generic UNIX LDAP server:

1. LDAP (Active Directory: Log in using Windows username @ domain): This choice pre-fillssettings that are suitable for logging in using your Windows user-name or Windows user-namefollowed by @ and the domain name.

2. LDAP (Active Directory: Log in using email address): This choice pre-fills settings that aresuitable for logging in with your email address.

3. LDAP (Generic: Log in using username @ domain): This choice pre-fills settings that aresuitable for logging in with your user-id (or user-id followed by @ and domain name.)

4. LDAP (Generic: Log in using email address): This choice pre-fills settings that are suitable forlogging in with your email address.

Note: Once an LDAP user lookup is created, editing it shows the method as simply LDAP. The four possi-bilities enumerated above are simply conveniences that pre-select appropriate settings when you firstcreate the user lookup.

LDAP user lookups can be used for one or both of user authentication and stream mapping. When usedfor stream mapping, the LDAP lookup method will also validate incoming email addresses against theLDAP server, allowing rejection of invalid recipients immediately at the CanIt gateway.

Note: In order for the LDAP User Lookup to validate incoming recipient addresses, it must be used forstreaming in Domain Mapping. Be sure to use another method of validation (e.g. Verification Servers(see Section 5.4, Valid Recipients table) if you do not use your User Lookup for streaming.



If you select one of the LDAP methods, you will see the LDAP User Lookup Wizard:

Figure 7.5: LDAP User Lookup

To complete the setup:

• In the “LDAP server(s)” box, enter the IP address or fully-qualified host name of your LDAPserver. You can enter a comma-separated list of servers if you have more than one LDAPserver. As with the IMAP and POP3 User Lookups, if a server listens on a non-standard port,



enter a slash followed by the port number after the server name. For example, if you have twoLDAP servers serverA and serverB, and the second listens on non-standard port 3389, enterthe following into the server box:

serverA, serverB/3389

If you want to use LDAPS (LDAP over SSL), enter the host name as an “ldaps” URL. Forexample:

ldaps://server.example.com/

• Enter the Base DN of your LDAP tree in the “Base DN” box.

• Typically, CanIt-Domain-PRO needs to bind to the LDAP directory before it can search it.Enter the Bind DN in the “Bind DN” box. If a password is required, enter it in the “Bindpassword” box. Note that Active Directory does not support anonymous bind; a Bind DN andBind password are required.

• If you wish to use this User Lookup for authentication, set “Use this method for authentication?”to Yes.

• If you would like to strip the domain name from the login name before attempting authentica-tion, set the “Strip domain name” setting to Yes. If someone logs in to CanIt-Domain-PRO [email protected] and this setting is Yes, then the username passed to the LDAP server issimply user.

• If you would like CanIt-Domain-PRO to force user-names authenticated by LDAP to lower-case, set “Force user name to lower-case” to Yes. (This also implicitly sets the home streamname on login to lower-case.) The user name is lower-cased before being presented to theLDAP server.

• Enter the search filter for login authentication. The string %s will be replaced by the user’s loginname. For most UNIX LDAP servers, a search filter of (uid=%s) is appropriate. For ActiveDirectory, it might be (sAMAccountName=%s).

• To use the Locked Addresses feature, CanIt-Domain-PRO needs to know the e-mail address ofa logged-in user. In most UNIX LDAP servers, this is stored in the mail attribute, while inmany Active Directory servers, this is stored in the attribute proxyAddresses. Enter theappropriate value in “Attribute containing user’s e-mail address”.

• If you wish to control group membership using LDAP, enter the name of an LDAP attribute inthe “Attribute containing group names” box. This attribute should contain a comma-separatedlist of group names. When a user authenticates, he/she will be considered to be a member of allof the groups listed in this attribute.

• By default, when a user successfully logs in via LDAP, CanIt-Domain-PRO caches the user-name and encrypted password for 5 days. If your LDAP server ever goes down, this permitsusers to continue to log in to CanIt-Domain-PRO (provided they have logged in successfullywithin the past 5 days.) You can change the cache time by editing “Number of days to cachesuccessful credentials”. If you set this parameter to zero, then CanIt-Domain-PRO will notcache credentials upon successful login.



• If you wish to use the LDAP server to stream addresses as well as authenticate, set “Use thismethod for streaming” to Yes.

• Enter the “Search filter for streaming”. For streaming, CanIt-Domain-PRO needs to look upan e-mail address in the LDAP server. For most UNIX servers, the appropriate search filter is(mail=%s), while for Active Directory, it is probably (proxyAddresses=smtp:%s). Inthe search filter, the string %s is replaced with the e-mail address. %u is replaced with the localpart of the e-mail address (everything before ‘@’) and %d is replaced with the domain part ofthe address (everything after the ‘@’.)

• If you would like CanIt-Domain-PRO to force stream names (as determined by the LDAPlookup) to lower-case, set “Force stream name to lower-case?” to Yes. (This is the default.)If you want to preserve mixed-case stream names, set this setting to No.

• CanIt-Domain-PRO needs to know which LDAP attribute contains the stream name. For mostUNIX servers, the appropriate attribute is uid, while for Active Directory, it is probablysAMAccountName. You can use a comma-separated list of attribute names for the “List ofattributes to user for stream name” entry. CanIt-Domain-PRO will examine the attributes inorder and set the stream name to the first attribute found that exists and is non-blank. This isuseful if not all of your LDAP objects contain the same set of attributes, but they all contain atleast one attribute appropriate for use as the stream name.

• If CanIt-Domain-PRO successfully looks up an e-mail address, but the LDAP record lacks anattribute for the stream name, CanIt-Domain-PRO can take one of the following actions:

– Tempfail the mail. We do not recommend this choice; it is available only for backward-compatibility with earlier versions of CanIt-Domain-PRO.

– Place the mail in the default stream.

– Place the mail in a stream whose name is the same as the entire email address. This issimilar to AsIs address mapping. In this case, mail to [email protected] will go intoa stream called [email protected].

– Place the mail in a stream whose name is the user-part of the email address. This is similarto ChopDomain address mapping. In this case, mail to [email protected] will gointo a stream called user.

– Place the mail in a stream whose name is the domain-part of the email address. This issimilar to ChopUser address mapping. In this case, mail to [email protected] willgo into a stream called example.org.

Set “Action if stream attribute missing” to the choice that is appropriate for your organization.

Note: Recipient Validation (i.e. rejecting SMTP RCPT with ”User Unknown” when the address is notfound in LDAP) is only done if CanIt-Domain-PRO receives an actual response that there is nocorresponding LDAP record for the given e-mail address. Changes to this setting do not affectvalidation.

Note: If the LDAP lookup for an email address returns more than one stream (because multiple LDAPentries match the address, for example), then CanIt-Domain-PRO picks a stream using the “Ac-



tion if stream attribute missing” setting. It also raises an anomaly since this is usually a seriouserror in the LDAP data; a given email address should be owned by one and only one stream.

• Normally, CanIt-Domain-PRO tries the LDAP servers in order. If you would like it to try themin a random order (for load-balancing), set “Load-balance LDAP servers” to Yes.

• Some LDAP servers require CanIt-Domain-PRO to disconnect and reconnect and re-bind be-tween queries. (Active Directory requires this.) If your LDAP server requires this, set the“Reconnect for additional queries” setting to Yes.

• If you would like CanIt-Domain-PRO to cache stream lookups, set “Cache stream lookups indatabase” to Yes.

• You can change the connect timeout from the default value of 120 seconds to any value from2 to 120 seconds. This timeout only applies to streaming lookups by the Perl filters. It doesnot apply to authentication, because PHP (used for the Web interface) does not have a way tospecify an LDAP connect timeout.

Once you have entered the LDAP parameters, click Next to review your entries, and Finish to createthe User Lookup.

7.2.3 Azure Active Directory Streaming

CanIt-Domain-PRO can perform directory lookups against Microsoft’s cloud-based Azure Active Di-rectory for the purpose of streaming. Integrating with Azure Active Directory requires setup steps tobe performed both in Azure and in CanIt-Domain-PRO.

Configuration within Azure

To integrate CanIt-Domain-PRO with Azure Active Directory, log on to your Azure account. Thenfollow these steps:



• Click on the Azure Active Directory item in the main menu. The Azure Active Directoryscreen pops up:

Figure 7.6: Azure Active Directory Main Screen



• Click on App registrations and then create a new application. Use CanIt as the ap-plication name, Web app / API as the application type, and https://antispam.roaringpenguin.com/canit/ as the sign-on URL.

Figure 7.7: Azure Active Directory Application Registration


https://antispam.roaringpenguin.com/canit/

https://antispam.roaringpenguin.com/canit/


• Once the application has been created, edit its settings. Copy the Application ID somewheresafe; you will need to enter it into CanIt-Domain-PRO at a later date. In the example screenshot,the application ID is cc9cca77-851e-4212-b015-07b1254560cb.

Figure 7.8: Azure Active Directory Application Settings



• In the application settings screen, add API access for both the Windows Azure Active Di-rectory and Microsoft Graph APIs. In each case, select the appropriate API and then addpermissions.

Figure 7.9: Azure API Access Settings



• For each API that you add, enable the Read directory data permission in the ApplicationPermissions section.

Figure 7.10: Azure Read Directory Permission



• Go back to the Application Settings screen and create an API key. Use API as the descriptionand Never expires for the expiration date. After you create the key, Azure will display the valueof the key. Copy the key somewhere safe immediately. After this screen is refreshed, there willbe no way to recover the key. The key will look like a random string of characters, somethinglike this: hkvSAkukD7GsUFYZKP7MdoZL0gJLpA+xrFBWGSj+zHY=

Figure 7.11: Azure API Key



Configuration within CanIt-Domain-PRO

Once you have created and configured the CanIt app within Azure and created an API key, you areready to configure Azure withing CanIt. Run the User Lookup creation wizard and pick Azure ActiveDirectory as the User Lookup Method. Click Next and the Azure Setup Page will appear:

Figure 7.12: Azure Setup within CanIt-Domain-PRO

Fill in the fields as follows:

• For Azure Tenant, use your Azure tenant name. This is typically the same as your domainname.

• For Application ID, use the Application ID you generated earlier within Azure.

• For Application Key, use the Application Key you generated earlier within Azure.

• The default Search Query looks for the login email address in the userPrinci-palName, the proxyAddresses and the otherMails fields. You may need to ad-just this query to suit your organization. The query syntax is described athttps://msdn.microsoft.com/en-us/library/azure/ad/graph/howto/


https://msdn.microsoft.com/en-us/library/azure/ad/graph/howto/azure-ad-graph-api-supported-queries-filters-and-paging-options#filter



azure-ad-graph-api-supported-queries-filters-and-paging-options#filter.

• The List of attributes to use for stream name is a comma-separated list of Azure AD at-tributes that are checked (in order) until a non-empty value is found. By default, we use theuserPrincipalName value as the stream name.

• The Action if stream attribute missing tells CanIt-Domain-PRO what do do if the search queryfinds an entry, but no stream attribute was found. This works exactly the same as with LDAPlookups described in Section 7.2.2.

• You should NOT change anything in the Advanced Settings section. The default Mi-crosoft logon URL is https://login.windows.net and the default Graph API URLis https://graph.windows.net. Overriding those URLs is strictly for Roaring Penguintest purposes.

7.2.4 Program Authentication and Streaming

With the Program User Lookup method, CanIt-Domain-PRO invokes an external program to authen-ticate users and map addresses to streams. If you select Program as your User Lookup type, theProgram User Lookup Wizard appears:

Figure 7.13: Program User Lookup

To configure the Program User Lookup:

• Enter the full path to your “account-info” script. This is an executable script or program thatyou must supply. The path you supply must be an absolute path name. If you are running aCanIt-Domain-PRO cluster, this script must exist (and be identical!) on all scanning servers andthe Web server.

• If you would like to strip the domain name from the login name before attempting authenti-cation, set the “Strip domain name” setting to Yes. If someone logs in to CanIt-Domain-PROas [email protected] and this setting is Yes, then the username passed to the program issimply user. The home stream, however, is normally [email protected].

• If you would like to strip the domain name from the home stream, set “Strip domain namefrom home stream after authentication?” to Yes. This means that if someone logs in [email protected], her home stream will be user.






• If you would like to cache stream lookups, set “Cache stream lookups in database?” to Yes. Westrongly recommend enabling caching.

How the Program User Lookup is Invoked

• For authentication, the program is invoked as follows:

/path/to/script --authenticate

The program is then expected to read two lines from its standard input: The first line is a loginname, and the second line is a password. The program must then validate the login name andpassword, and exit with one of the following exit codes:

– 0 — Authentication was successful.

– 1 — Authentication failed.

• For obtaining user information, the program is invoked as follows:

/path/to/script --info username

Here, the program is passed the successfully logged-on user name as a command-line argument.It should print a series of key=value lines to its standard output, and exit with an exit status of0. (The script doesn’t have to produce any output, but it can produce output if you want to passextra information to CanIt-Domain-PRO.)

The key/value pairs currently used by CanIt-Domain-PRO are:

– home stream=stream-name — sets the user’s home stream to stream-name insteadof his or her login name. One possible use could be to convert a login name to all lower-case on systems that permit case-insensitive authentication. This ensures that no matterhow the person logs in, she is directed to the correct stream name.

– groups=group1,group2,...,groupN — when the user logs in, add her to all ofthe groups listed in the comma-separated list.

– mail=email-address — set the user’s e-mail address to email-address.

• For mapping an e-mail address to a stream, the program is invoked as follows:

/path/to/script --info-email address

Here, address is an e-mail address that must be streamed. The script should write key=valuelines to its standard output, and exit with one of the following exit codes:

– 0 — the address exists and was successfully streamed.

– 1 — there was a temporary failure streaming the address. The mail will be tempfailed.

– 67 — the address is not valid. CanIt-Domain-PRO will fail the SMTP RCPT commandwith a “User unknown” failure code.

If the address was streamed successfully, the script must print the following line to standardoutput:

stream=stream-name



This causes address to be mapped to stream-name. If no stream=stream-name line isemitted, but the script exits with a zero status, then CanIt-Domain-PRO falls back to databaselookups, as described in Section 2.5 on page 33.

Sample Program for the Program User Lookup Method

The following is a very simple Bourne shell script illustrating how the Program User Lookup methodworks. Real scripts would obviously be more complex and probably written in a more appropriatelanguage like Perl.



#!/bin/shdo_auth () {

read userread pass# In reality, we would do a directory lookup against LDAP or similarif test "$user" = "foo" -a "$pass" = "bar" ; then

exit 0fiexit 1

}

do_info () {user="$1"# In reality, we would do a directory lookup against LDAP or similarif test "$user" = "foo" ; then

echo "home_stream=foobar";echo "[email protected]";

fiexit 0

}

do_info_email () {email="$1"# In reality, we would do a directory lookup against LDAP or similarif test "$email" = "[email protected]" ; then

echo "stream=foobar-stream";fiif test "$email" = "[email protected]" ; then

# No such userexit 67

fiexit 0

}

# Main programcase "$1" in

--authenticate)do_auth;;

--info)do_info "$2";;

--info-email)do_info_email "$2";;

*)exit 1;;;

esac



7.2.5 Program Authentication (Legacy Method)

If you select this User Lookup method, then CanIt-Domain-PRO falls back to behavior compatiblewith previous versions. (This behavior is deprecated, however. New installations should use ProgramAuthentication as described in Section 7.2.4.)

• If a program called /usr/share/canit/scripts/account-info exists and is exe-cutable, CanIt-Domain-PRO invokes it as if it were the script supplied for a Program UserLookup method.

• Otherwise, CanIt-Domain-PRO invokes /usr/share/canit/scripts/authenticate-userto authenticate users and /usr/share/canit/scripts/address-to-stream toconvert an e-mail address to a stream. These scripts have been in use since CanIt-Domain-PRO2.0 and are deprecated; you should convert to the new Program User Lookup method.

7.2.6 The account-info Script

Some User Lookup methods (such as POP3 or IMAP) as well as a lookup in the built-in user databaseare not capable of passing extra information back to CanIt-Domain-PRO. For that reason, if any UserLookup method other than Program or LDAP is used, CanIt-Domain-PRO still attempts to execute:

/usr/share/canit/scripts/account-info --info username

to obtain extra attributes (mail, groups and home stream) after a user logs in. If you need to setusers’ e-mail addresses or home streams, but have them authenticate against an IMAP or POP3 server,simply supply an appropriate account-info script.

7.2.7 The Rewrite User Lookup

The rewrite user lookup is not used for authentication. It is only used to convert an address to a stream.It does so by rewriting the email address using a rewrite expression.

To create a Rewrite User Lookup, enter the rewrite expression. CanIt-Domain-PRO rewrites an ad-dress as follows:

• The sequence %u in the rewrite expression is replaced with the local part of the email address(that is, everything before the @ sign.)

• The sequence %d in the rewrite expression is replaced with the domain part of the email address(that is, everything after the @ sign.)

• The sequence %s in the rewrite expression is replaced with the entire email address.

• Any other characters in the rewrite expression are copied as-is.

As an example of how you’d use the rewrite user lookup, consider an organization that ownsthe domains example.com, example.org and example.net. It wants any email address



user@example.* to be placed in the stream [email protected]. That is, no matter whatthe domain on the incoming email address, it should be replaced with example.com.

This can be accomplished by creating a rewrite user lookup with a rewrite expression of:

%[email protected]

and then using that user lookup as the Domain Mapping entry (Section 5.14) for all of the domainsexample.com, example.net and example.org

Finally, observe that the Rewrite User Lookup can implement AsIs, ChopDomain and ChopUserstreaming (Section 5.14). The relevant rewrite expressions are:

• AsIs: %u@%d

• ChopDomain: %u

• ChopUser: %d

7.3 Authentication Mappings

Once you have set up your User Lookup methods, you need to tell CanIt-Domain-PRO which methodto invoke for each domain. To do this, click on Setup and then Authentication Mappings. TheAuthentication Mappings page appears:

Figure 7.14: Authentication Mappings

To create a new authentication mapping:

1. Enter the domain name in the Domain field. If you enter a single asterisk (“*”) in this field,then it is used as the default authentication mapping if an exact match is not found.

2. Select the User Lookup from the Mapping field. If there are any User Lookup methods addedto ancestor realms of the current realm, they will appear as additional choices if they are markedas being available for subrealms.


7.3. AUTHENTICATION MAPPINGS 159

3. Click on Submit Changes

In Figure 7.14, we see that anyone who logs in as [email protected] will be authen-ticated using the POP3-Sample User Lookup. Anyone logging in with a different domain (or nodomain at all—simply user) will be authenticated using the LDAP-Sample User Lookup.

If you enter a string in the “Filter:” box, then CanIt-Domain-PRO limits the display to entries whoseDomain or Mapping columns contain that string.


Chapter 8

Bayesian Filtering

8.1 Introduction to Bayesian Filtering

Bayesian filtering is a statistical technique whereby CanIt-Domain-PRO assigns a spam probabilitybased on training from users. Bayesian filtering can greatly improve the accuracy of CanIt-Domain-PRO, and makes it harder for spammers to evade filtering.

Please consult the CanIt-Domain-PRO User’s Guide for additional details on using Bayesian filtering.This guide only contains information relevant when setting up and administering CanIt-Domain-PRO.

8.2 Unauthenticated Voting

Normally, to vote if a message is spam or not spam, a user must log in. You can configure CanIt-Domain-PRO to permit unauthenticated voting; this can make life easier for end-users who can justclick on a link without worrying about entering a user name and password.

Note: Think carefully about permitting unauthenticated voting. If voting links ever escape your organization(as part of a forwarded message, for example), and your CanIt-Domain-PRO Web interface is exter-nally accessible, outsiders can cast votes. We strongly recommend permitting unauthenticated votingonly if access the the CanIt-Domain-PRO Web interface is controlled in some other way.

To permit unauthenticated voting:

• Under Preferences and Quarantine Settings, set Permit unauthenticated voting to Yes

You can permit unauthenticated voting on a stream-by-stream basis. If you permit it in the defaultstream, then it will be permitted in all streams that inherit from default (and that don’t override thesetting.)


162 CHAPTER 8. BAYESIAN FILTERING

8.3 The Bayes Journal

Bayesian training can be slow because it involves many database updates. For that reason, when youtrain a message, CanIt-Domain-PRO simply makes a note of the fact that the message is to be trainedin a special table called the Bayes Journal. Periodically, a CanIt-Domain-PRO daemon process goesthrough the Bayes Journal and actually updates the Bayes data.

For this reason, if you train some messages, these results will not immediately appear in the BayesSettings page. The Bayes Journal is run every 10 minutes or so, so your training should appear within10-15 minutes.

8.4 Site-Wide and Realm-Wide Bayes Training

Whenever someone hand-trains a message, the message is trained in the default stream of the realmas well as the stream containing the message. Additionally, it is trained in the default stream of allancestor realms. For example, if the realm foo is a subrealm of base and the realm bar is a subrealmof foo, then hand-training a message in the stream bar:quux also trains it in bar:default,foo:default and base:default. You may wish to add some or all of these ancestor-realmdefault streams to the list of streams from which Bayes training is inherited.

8.5 RPTN

RPTN stands for the Roaring Penguin Training Network, and is a mechanism whereby multiple CanItinstallations can share Bayes votes. RPTN contains two main parts:

1. In the reporting phase, CanIt-Domain-PRO installations send reports about whether or not mailthey have seen is spam. A report essentially consists of a list of tokens in the mail message anda spam or not-spam flag, depending on how the incident was disposed of. The RPTN serveraggregates all of the reports it receives and builds a database of Bayesian statistics from thereports.

2. In the download phase, a CanIt-Domain-PRO installation downloads the aggregated data andinstalls it in its database. This data can subsequently be used for Bayesian analysis.

To set up RPTN, click on Setup and then Wizards. Choose the RPTN Setup Wizard. The wizardleads you through the following steps:

1. You are asked if you would like to download Bayes data from RPTN.

2. If you answered Yes in Step 1, you are given an opportunity to limit when RPTN data is down-loaded. Downloading RPTN data can place a fair amount of load on the server, so you shouldlimit RPTN downloads to off-peak hours. Be sure to leave at least a four-hour download win-dow, because RPTN checks are made every two hours. If the download window is too short,you may miss a download.


8.6. RULESET AND GEOLOCATION DATA UPDATES 163

3. You are asked if you would like to submit reports to RPTN.

4. If you answered Yes in steps 1 or 3, you are prompted for your download username and pass-word. You cannot submit RPTN reports or download RPTN data unless you supply a validusername and password.

5. Your settings are summarized, and you are prompted to click Finish to save the changes.

RPTN data are downloaded into a stream called @@RPTN. If you would like to use RPTN data inBayesian analysis, you must include @@RPTN in the stream setting “Inherit Bayes training historyfrom these streams”. If you want all streams to inherit Bayes data from @@RPTN, then set the “InheritBayes training history from these streams” setting in the default stream in the base realm.

Note: To download RPTN data, the CanIt-Domain-PRO server must be able to make outgoing HTTPS con-nections (over TCP port 443) to the machine server.rptn.ca. To submit RPTN reports, theserver must be able to make outgoing HTTPS connections to server.rptn.ca and also be per-mitted to send outgoing e-mail to [email protected]. If you have a firewall in front of theCanIt-Domain-PRO server, please ensure that the firewall rules permit the RPTN traffic.

8.6 Ruleset and Geolocation Data Updates

In addition to downloading Bayes data, CanIt-Domain-PRO uses your RPTN credentials to downloadtwo other sets of data:

• Updated rules that are pushed out from time-to-time by Roaring Penguin.

• Geolocation data that maps IP addresses to countries and cities. (The data are derived fromthe GeoLite City data from MaxMind, which requires the following acknowledgement: Thisproduct includes GeoLite data created by MaxMind, available from http://www.maxmind.com/)

The updated rulesets are simply SpamAssassin rules that Roaring Penguin publishes as required whena new spam variant is detected. The geolocation data is used by the country rules as described in theUser’s Guide. CanIt-Domain-PRO also tokenizes the country, region, city and latitude/longitude ofthe sending relay for use in the Bayes database.


164 CHAPTER 8. BAYESIAN FILTERING


Chapter 9

Permissions

9.1 Introduction

In addition to the fairly coarse-grained settings described in Section 6.5.1, “User Privileges”, CanIt-Domain-PRO allows you to implement fine-grained control over access to various parts of the Web-based interface.

CanIt-Domain-PRO has two kinds of permissions:

1. Stream Permissions control access to CanIt-Domain-PRO features that affect the filtering of e-mail. For example, the ability to allow or block senders, create custom rules, and so on are allStream Permissions. Stream Permissions depend on both the user and the stream; a given usermay have different permissions in different streams.

2. User Permissions control access to various parts of the CanIt-Domain-PRO user-interface notdirectly connected to filtering mail. For example, access the different GUI preferences and theability to do WHOIS lookups are all User Permissions.

CanIt-Domain-PRO can associate permissions with users and with groups. Any user can be a memberof zero or more groups. CanIt-Domain-PRO always grants a user the union of all his user-specificpermissions and all his group permissions. Adding a user to a group, therefore, can only ever grantadditional permissions. It cannot take away permissions.

9.2 Stream Permissions

Every stream has associated with it an ordered list of stream classes. When CanIt-Domain-PRO looksup stream permissions, it first calculates the list of stream classes associated with a particular user andstream. Here is how CanIt-Domain-PRO computes the list of stream classes:

1. The name of the stream always comes first. Thus, for example, if you are viewing a streamcalled mystream, then the list of stream classes starts with mystream.


166 CHAPTER 9. PERMISSIONS

2. If mystream happens to be your “home stream” (Section 4.6), then @@HOME is added to thelist of stream classes.

3. If you have write-access in mystream, then @@WRITABLE is added to the list of streamclasses.

4. If you have read-access in mystream, then @@READABLE is added to the list of stream classes.

5. Finally, the wildcard value * is added to the end of the list of stream classes.

When CanIt-Domain-PRO determines what permissions you have in a particular stream, it uses thefollowing procedure:

1. It looks for permissions granted in the actual stream name. If it finds any, it stops searching thestream classes.

2. Otherwise, it checks the the stream classes and adds all permissions found to the set of grantedpermissions.

9.3 Determining Permissions

To determine a particular user’s permissions, CanIt-Domain-PRO performs the following steps:

1. First, it gathers all permissions associated with the particular user’s login ID. (These permissionsare shown in Figures 9.3 and 9.4.)

2. Next, it adds all permissions granted to all the groups to which the user belongs.

3. If there was no entry in the permissions table for the particular user (that is, if Step 1 found noentries), then CanIt-Domain-PRO performs the following steps:

(a) If the user has root privileges, then CanIt-Domain-PRO adds all permissions granted tothe pseudo-user *root* or *localroot* in the user’s realm.

(b) Next, CanIt-Domain-PRO adds all permissions granted to the wild-card user * in the user’srealm.

4. If no entry was found for Step 3, then CanIt-Domain-PRO performs the following steps:

(a) If the user has root privileges, then CanIt-Domain-PRO adds all permissions granted tothe pseudo-user *root* found in the first ancestor realm encountered on the path up tobase.

(b) Next, CanIt-Domain-PRO adds all permissions granted to the wild-card user * in the firstancestor realm encountered on the path up to base.


9.4. GRANTING PERMISSIONS 167

9.4 Granting Permissions

To grant or deny permissions, click on Administration and then Permissions. The Permissions Pageappears:

Figure 9.1: Permissions Page

If you enter a string in the “Filter:” box, then CanIt-Domain-PRO limits the display to entries whoseUser column contains that string.

If you want to edit permissions for groups rather than users, click on the Groups link:

Figure 9.2: Permissions Page

9.4.1 Granting Stream Permissions

To grant stream permissions, click on the Edit link in the Stream Permissions column. The StreamPermissions page appears:



Figure 9.3: Stream Permissions Page

• To enable a stream permission in a particular stream or stream class, enable the checkbox in theappropriate row and column.

• To enter the name of a stream or stream class, enter it into the text box in the Per-StreamPermission row. Note that when you enter permissions for a new user, you must enter thestream class in the text box, or your changes will be discarded.

• To delete all permissions for a particular stream or stream class, click the Delete link at thebottom of the appropriate column.

• To view permissions only for one stream or stream class, click on the stream or stream classname.

• To make your changes take effect, click Submit Changes.

The Stream Permissions are:

• Block Senders – The user is permitted to block senders.

• Always Allow Senders – The user is permitted to always-allow senders.

• Hold/Tag Senders – The user is permitted to add a hold rule for senders.

• Block/Always Allow/Hold/Tag Domains – These permissions are similar to the Sender Actionpermissions, but they apply to domain rules.

• Block/Always Allow/Hold/Tag Networks – These permissions are similar to the Sender Actionpermissions, but they apply to network rules.

• Reject/Accept/Hold/Tag MIME Types – These permissions are similar to the Sender Actionpermissions, but they apply to MIME type rules.

• Reject/Accept/Hold/Tag Filename Extensions – These permissions are similar to the SenderAction permissions, but they apply to filename extension rules.


9.4. GRANTING PERMISSIONS 169

• Custom Rules – The user is permitted to create custom rules.

• SPF Rules – The user is permitted to create SPF rules.

• RBL Rules – The user is permitted to create RBL rules.

• Country Rules – The user is permitted to create country-code rules.

• Bayes Settings – The user is permitted to edit Bayes scoring rules.

• Blocked Recipients – The user can block recipients.

• Valid Recipients – The user can enter recipients into the Valid Recipients Table.

• See Pending/Non-Spam/Spam Message – The user can see the specified message type in thequarantine. Note that these permissions are normally off for @@READABLE streams; otherwise,the user could see default’s spam quarantine.

• Add Alternate Addresses to Streams – The user can add aliases to his/her stream.

• Opt In/Out – The user can opt in or out of spam-scanning.

• Adjust Notification Settings – The user can adjust his or her notification settings.

• See Per-Stream/Global Reports – The user can see the specified reports.

• Quarantine Settings – Every quarantine setting has an associated permission. The user canonly see a quarantine setting if its corresponding permission has been granted. The user canonly change a quarantine setting if the permission has been granted and the user has write-access in the stream.

Note: If a user does not have write-access in a stream, then permissions such as Custom Rules, AlwaysAllow Senders, etc. merely permit the user to see the rules. He or she still cannot change them.

9.4.2 Granting User Permissions

To grant user permissions, click on the Edit link in the User Permissions column. The User Permis-sions page appears:



Figure 9.4: User Permissions Page

The following User Permissions may be granted:

• Preferences – Unless this permission is granted, the user will not have access to the Preferencesmenu or any of its sub-menus.

• WHOIS Lookups – If this permission is granted, the user will be allowed to do WHOISlookups.

• See Statistics – Allows the user to see the Reports : Statistics page.

• Use Log Searching – Allows the user to use the Log Searching feature (Chapter 17).

Note: Users must have root privileges to use Log Searching; non-root users cannot use it even if UseLog Searching is enabled. Also, the log-searching feature is available only on CanIt-Domain-PRO appliances.

• See User’s Guide – Enables the link to the user’s guide.

• Use API – Allows the user to access the REST-based CanIt-Domain-PRO API. See the APIGuide for details.

• Provision Domains via API – Allows the user to provision new realms and domains via theAPI. Note that a user must be a realm administrator and must have API access to be able toprovision domains. This option is available only on our CanIt-Domain-PRO appliances andHosted CanIt.

Note: Allowing users to provision new realms and domains grants them tremendous power and maybe a security risk. Do not grant this permission except to highly-trusted realm administrators

• Use Expert Interface – Grants the user access to the expert interface.

• Create RSS Feed – Grants the user permission to create an RSS feed link for pending messages.


9.5. PERMISSION GRANTABILITY 171

• Turn off Stream Inheritance – Grants the user permission to completely isolate his stream bydisabling inheritance from the default stream. We do not recommend granting this permissionas a matter of course.

• Preferences – Each preference setting has an associated permission. A user can only changethose settings for which permission has been granted.

9.5 Permission Grantability

In CanIt-Domain-PRO, the System Administrator always has all permissions and can grant or deny allpermissions. However, the System Administrator can both limit the permissions available to RealmAdministrators (as described in Section 9.4) and limit which permissions Realm Administrators cangrant to themselves and other users.

To modify which permissions realm administrators can grant, click on Administration and then Per-mission Grantability. The Grantable Permissions page appears:

Figure 9.5: Permission Grantability

To delete an entry from the Permission Grantability table, check the Delete checkbox and click SubmitChanges.

To edit which permissions a user can grant, click on the Edit link in the appropriate table row. To adda user to the table, enter the user ID in the User box and click Add. You can specify a realm in theRealm pulldown; if you do not, then CanIt-Domain-PRO will determine the realm based on the userID.

Whether you add a new user or edit an existing user, CanIt-Domain-PRO brings up the GrantablePermissions Detail page:



Figure 9.6: Grantable Permissions Detail

To allow a user to grant a permission, check the appropriate checkbox. To prevent a user from grantinga permission, uncheck the checkbox. Click Submit Changes when you have set permission grantabil-ity as you desire.

Note: If you prevent a user from granting a permission, you should also turn off that permission for the user.Otherwise, the user will lose the permission if he/she ever updates his permissions. For example, if arealm administrator is permitted to block senders, but not allowed to grant that permission, then if sheever modifies her own permissions, she will lose the “Block Sender” permission.

9.5.1 Grantability Algorithm

CanIt-Domain-PRO uses the following algorithm to determine which permissions a realm admin-istrator can grant. For the sake of illustration, assume that the realm administrator’s user ID [email protected] and the realm name is myrealm.

1. First, CanIt-Domain-PRO looks for a grantability entry specifically for [email protected] the realm myrealm. If it finds such an entry, it uses it.

2. If Step 1 found no entry, CanIt-Domain-PRO looks for an entry for the user * in the realmmyrealm.

3. If Step 2 found no entry, CanIt-Domain-PRO looks for an entry for the user * in the realmbase.

4. If Step 3 found no entry, then the realm administrator is allowed to grant any permission.


Chapter 10

Streams, Inheritance and the SimpleGUI

10.1 Simplification

CanIt-Domain-PRO is extremely versatile, allowing end-users to set many parameters such as blockrules, allow rules, custom rules, and so on. For many users, this is intimidating—the users may beunsophisticated, and just want to “make spam stop.”

CanIt-Domain-PRO allows the administrator to set up special streams with pre-configured settings.Unsophisticated users then see a very simple interface which allows them to choose from one of thesesettings. CanIt-Domain-PRO achieves this with stream inheritance and special streams.

Note: Users who use the Simple GUI will not have their own quarantines. Special streams should be con-figured to pass, tag or reject. If any incidents are actually created, someone with administrative accesswill need to check the special streams’ quarantines periodically.

10.2 Stream Inheritance

Streams in CanIt-Domain-PRO inherit rules and settings from other streams. By default, all streamsin a given realm inherit rules and settings from the default stream in that realm. The defaultstream, in turn, inherits rules and settings from the default stream in the parent realm and so on allthe way up to the base realm.

If a stream stream1 inherits from another stream stream2, we refer to stream2 as the parent ofstream1. Conversely, we call stream1 the child of stream2.

Furthermore, suppose that stream2 inherits from stream3. We then call stream3 and stream2the ancestors of stream1. These terms are illustrated in Figure 10.1:


174 CHAPTER 10. STREAMS, INHERITANCE AND THE SIMPLE GUI

stream1 inherits from stream2

stream1 is the child of stream2

stream2 is the parent of stream1

stream3 and stream2 are the ancestors of stream1

stream2 inherits from stream3

stream1

stream2

stream3

parent

child

child

parent

Figure 10.1: Stream Inheritance Terminology

In addition to the default inheritance, streams can be configured to inherit rules and settings fromSpecial Streams (discussed next in Section 10.3.)

To determine a stream’s inheritance, CanIt-Domain-PRO consults the Stream Inheritance Table. Tosee this table, click on Administration and then Inheritance:

Figure 10.2: Stream Inheritance Table

To determine a stream’s parent, CanIt-Domain-PRO first looks up the stream in the inheritance table.If there is an entry, then that entry is used to determine the parent. If there was no entry, CanIt-Domain-PRO looks up the key “*” in the inheritance table. If such an entry exists, it is used to determine theparent.

In the example in Figure 10.2:

• user3 inherits from 01 Tag Only.

• user4 inherits from 00 Opt Out.

• user5 does not inherit from any other stream.

• user9 inherits from default.


10.3. SPECIAL STREAMS 175

• All other streams (except for default) inherit from 01 Tag Only, because of the wildcardentry.

If you enter a string in the “Filter:” box, then CanIt-Domain-PRO limits the display to entries whoseStream or Inherits From columns contain that string.

10.3 Special Streams

A Special Stream is a normal stream with two extra behaviors:

• Other streams are allowed to inherit from special streams. Normally, a stream can only havedefault as its parent. If you add special streams, however, other streams are allowed to makethe special streams their parents.

• If a stream inherits from a special stream, then mail for the child stream is held in the parent’squarantine. That is, by inheriting from a special stream, a stream “loses” its quarantine, givingresponsibility for any quarantined mail to the special stream.

10.3.1 Final Streams

A special stream may be marked final. If a special stream is marked final, then children of that streammay not override the special stream’s rules or settings. If a stream inherits from a final special stream,it’s as if the stream has given all control over to the special stream.

To see special streams, click on Administration and then Special Streams. The Special Stream Tableappears:

Figure 10.3: Special Stream Table

10.3.2 Creating Special Streams

To create a special stream, enter the name of the stream in the Stream text box, and a user-friendlydescription in the Description box. Then click Add Special Stream.



In the example, the four streams 00 Opt Out, 10 Tag Only, 20 IT Staff and30 Aggressive have been created. (Special streams are presented to end-users in order ofthe stream name, so we named the streams beginning with numbers so they would sort from leastto most aggressive. We leave gaps between the stream numbers so we can insert more streams inbetween if required.)

Once you have created the special streams, configure them appropriately. For example, for00 Opt Out, you’d switch into that stream, and then under Preferences : Opt In/Out, you’d optthat stream out. (For convenience, you can click on a stream name in the Special Stream Table toswitch into that stream.) For 30 Aggressive, you might change the stream settings to auto-discardanything scoring 8 or more on the spam scale. For 20 IT Staff, you could have CanIt-Domain-PRO hold suspect spam, and have a member of your IT staff check 20 IT Staff’s quarantine andrelease false-positives.

Note that 00 Opt Out and 20 IT Staff are marked final. This means that rules and settings instreams inheriting from these two special streams are ignored; only the special streams’ settings andrules are used. On the other hand, streams inheriting from 10 Tag Only and 30 Aggressivemaydefine their own rules, settings, block rules and allow rules.

You can define as many special streams with as many different settings as you deem appropriate. Notethat all special streams (by default) inherit from the default stream.

10.3.3 Deleting Special Streams

To delete a special stream, enable the checkbox in the Delete? column for the appropriate stream.Then click Submit Changes. Warning: If you delete a special stream, then all inheritances from thatstream are deleted. Please see Section 10.2 for more details.

10.4 The Simplified GUI

If the CanIt-Domain-PRO administrator enabled the global setting G-4060 Users authenticated byalternate means default to simple GUI? (Section 6.1), then such users only see the Simplified Inter-face:

Figure 10.4: Simplified Interface


10.5. INHERITANCE FROM NON-FINAL STREAMS 177

The simplified interface simply lists the possible Special Streams. The currently-inherited specialstream is highlighted in bold red print.

To inherit from a different stream, the user simply clicks on the appropriate radio button and clicksSet Spam-Scanning Level. This adjusts the entry in the inheritance table.

To log out, the user clicks on Log Out.If the user clicks on Enable Expert Interface, then he or she will have access to the usual CanIt-Domain-PRO interface. He or she can then turn off inheritance (via Preferences : Set DefaultStream) and take control over his or her own block and allow rules and spam quarantine.

Note: If you have set the global setting G-4075 Switching to expert mode cancels stream inheritanceto Yes, then the act of clicking Enable Expert Interface cancels any inheritance that was in force,making the stream inherit from default again.

To get back to the simple GUI, click on Simple Interface top-level menu entry. Note that this menuentry does not appear until at least one special stream has been defined.

10.5 Inheritance from Non-Final Streams

If a stream inherits from a non-final stream, CanIt-Domain-PRO uses the following procedures toresolve rules. In these examples, we assume that stream john inherits from the non-final stream10 Tag Only

• For sender, domain and network block and allow rules, and for MIME type and Filename Ex-tension rules, CanIt-Domain-PRO first looks for a rule associated with the original stream (inour example, john.) If no such rule is found, it then tries the parent stream (in our example,10 Tag Only) and then the parent of the parent, and so on up the inheritance chain.

• For custom rules, CanIt-Domain-PRO uses all the rules associated with the original stream inaddition to rules associated with the ancestor streams.

• Bayes data is associated with the original stream (john) and not the parent stream(10 Tag Only).

10.6 Inheritance from Opted-Out Streams

If a stream or any of its ancestors is opted-out of spam-scanning, then no spam scanning is performed.


Chapter 11

Periodic Reports

11.1 Introduction

CanIt-Domain-PRO can generate PDF reports about mail filtering activity and e-mail them to specifiedrecipients.

11.1.1 Periodic Reports

A periodic report has a name, a page size, a recipient and a recurrence. The name can be anythingyou pick. The page size can be one of “US Letter” or “A4”. And the recipient can be any valid e-mailaddress. The recurrence specifies how often the report should be generated and mailed out. Possiblechoices for the recurrence are:

• On Demand — the report is never generated and mailed automatically, but only when specifi-cally requested from the Web interface.

• Daily — the report is generated and mailed daily.

• Weekly — the report is generated and mailed weekly. You can choose the day of the week.

• Monthly — the report is generated and mailed monthly. You can choose either the first orfifteenth day of the month.

11.1.2 Charts

A chart produces a single PDF page in a periodic report. It contains a chart corresponding to aparticular statistical query. A chart has a name (which can be anything you pick) and a type. Theavailable chart types are described below. Note that all charts accept parameters that modify theresults. For example, you can restrict the types of mail counted (you might only want to count spam,for example), the destination domains, etc.

In addition to producing a page in the PDF report, each chart also generates a CSV file for importinginto spreadsheet software. (Some charts only produce CSV files and no PDF output; if this is the case,it will be noted in the chart’s description.)


180 CHAPTER 11. PERIODIC REPORTS

• Classification of Recent Mail. A pie chart showing the breakdown of recently-received e-mail. (“Recent” e-mail is defined by Global Setting G-1550, “Number of hours to keep detailedstatistics”)

• Top Mail Countries. A pie chart showing the top countries sending recent e-mail.

• Top Domains. A pie chart showing the top recipient domains receiving recent e-mail.

• Top Mail Relays. A pie chart showing the top sending relays that have sent recent e-mail.

• Top Recipients. A pie chart showing the top recipient addresses receiving recent e-mail.

• Top Streams. A pie chart showing the top streams receiving recent e-mail.

• Top Viruses. A pie chart showing top recently-received viruses.

• Summary of Greylisting per Hour. A bar-chart showing how much recent e-mail wasgreylisted and ungreylisted.

• Summary of Mail per Hour. A bar-chart showing the classification of recent e-mail per hour.

• Classification of Long-Term Mail. A pie chart showing the breakdown of received e-mail overthe long term. The timespan available in long-term statistics is determined by Global SettingG-1500, “Expire statistics after this many days”.

• Top Domains (Long-Term Statistics). A pie chart showing the top recipient domains over thelong-term.

• Top Realms (Long-Term Statistics). A pie chart showing the top recipient realms over thelong-term.

• Top Streams (Long-Term Statistics). A pie chart showing the top recipient streams over thelong-term.

• Summary of Greylisting per Day. A bar chart showing how much mail was greylisted andungreylisted over the long-term.

• Summary of Mail per Day. A bar chart showing how daily classification of mail over thelong-term.

• Summary of Mail per Realm per Day. A bar chart showing daily mail volume per realm overthe long term.

• Number of Email Addresses Seen by Realm. A chart showing the number of addresses seenin the last 30 days, broken down by realm. Note that this chart is only available as a CSV file; itdoes not produce PDF output.


11.2. CREATING CHARTS 181

11.2 Creating Charts

The first step in creating a periodic report is to create one or more charts.

Click on Reports : Periodic Reports. The main Periodic Reports page appears:

Figure 11.1: Periodic Reports

To add a chart:

1. Click Add a New Chart.

2. Enter a name for your chart. This name will appear as the page title in the final reports.

3. Select a chart type.

4. Click Next...

Once you have selected a chart type, CanIt-Domain-PRO will display a page for setting parametersfor the chart. Set the parameters as appropriate for your chart and click Save Chart.To edit an existing chart’s parameters, click on its name in the Name column.

To rename a chart, enter its new name in the Rename To... box and click Submit Changes.

To delete a chart, enable the corresponding checkbutton in the Delete... column and click SubmitChanges.

11.3 Creating Periodic Reports

Once you have created one or more charts, you can create periodic reports. To create a new periodicreport, click Add a New Report. The Add Periodic Report page appears:



Figure 11.2: Add Periodic Report

To create the report:

1. Pick a name for the report and enter it in the appropriate box.

2. Pick a time when the report should be sent. You can pick daily, weekly or monthly reports.You can also select “On-Demand Only”. Such reports are never sent automatically, but are onlygenerated on demand.

3. Enter an e-mail address to which the report should be sent. You can enter multiple addresses byseparating them with commas.

4. Select a page size for the report (A4 or US Letter).

5. Pick one or more charts for the report by enabling the appropriate Add checkboxes.

6. Click one of the Submit Changes buttons.

11.4 Editing Periodic Reports

To edit an existing periodic report, click on the report’s name in the Name column. You alter thereports parameters, add or remove charts, or move existing charts up or down from the report editingpage.

To delete a periodic report, enable the appropriate Delete... checkbox and click Submit Changes.


11.5. RUNNING A REPORT ON DEMAND 183

11.5 Running a Report on Demand

To run a specific periodic report on demand, enable the appropriate Run Now... checkbox and clickSubmit Changes. The report will be queued for processing. Note that it can take anywhere from afew minutes to a few hours for the report queue to be processed, so the report might take a while to bemailed out.


Chapter 12

Locked Addresses

12.1 Introduction to Locked Addresses

Locked Addresses are designed to solve the following problem: You want to give out your e-mailaddress to someone, but you don’t trust that person or organization not to turn around and give or sellit to others. You want an address that can only be used by the person or organization you give it to,and not by anyone else.

CanIt-Domain-PRO has a complete solution to this problem. However, it does require some adminis-trative overhead before users can take advantage of the feature.

12.2 Preparing to use Locked Addresses

Before end-users can use locked addresses, you need to perform the following steps.

12.2.1 Create a new domain

Choose a new domain, specifically for locked addresses. This domain should be a subdomain of your“real” domain. For example, if you own the domain roaringpenguin.com, you might chooseto place all your locked addresses in la.roaringpenguin.com. The domain you use for lockedaddresses should contain only locked addresses and should not be used for any “real” e-mail addresses.

12.2.2 Configure mail for the new domain

The next step is to configure the CanIt-Domain-PRO machine to receive mail for the new domain.Obviously, the first thing you need to do is publish an MX record for the domain. For example, if yourlocked address domain is la.roaringpenguin.com and your CanIt-Domain-PRO server’s nameis canit.roaringpenguin.com, you might add a DNS record that looks like this:

la.roaringpenguin.com. 1d IN MX 1 canit.roaringpenguin.com.


186 CHAPTER 12. LOCKED ADDRESSES

Also, you need to configure the CanIt-Domain-PRO machine to accept and discard all mail for thelocked domain. (Mail should never be delivered to addresses in the locked domain, but just in case,there should be a mechanism to discard them.)

Configuring Sendmail to accept mail for the locked domain is easy: Just add an entry in the accessdatabase. In our example, it would be:

To:la.roaringpenguin.com RELAY

(If you are running a CanIt-Domain-PRO Appliance, you can use Domain Routing from the Webinterface instead of manually editing Sendmail configuration files.)

The easiest way to configure Sendmail to discard mail for the locked domain is to make use of thevirtusertable feature. Add an entry like this in virtusertable:

@la.roaringpenguin.com [email protected]

and ensure that mail to [email protected] gets discarded (by making analias from devnull to /dev/null.)

Of course, you need to substitute your own locked address domain for la.roaringpenguin.comand your own CanIt-Domain-PRO server name for canit.roaringpenguin.com.)

12.2.3 Inform CanIt-Domain-PRO about the locked address domain

CanIt-Domain-PRO needs to know the domain you’re using for locked addresses, so it can treat anysuch addresses specially. In the Web interface, click on Administration : Global Settings and enterthe locked address domain into the global setting G-10000 “Domain for Locked Addresses”

12.2.4 Associate each login name with an e-mail address

CanIt-Domain-PRO can only generate locked addresses if it has a real e-mail address for each logged-in user. For users in CanIt-Domain-PRO’s built-in user table (Section 6.5 on page 116), simply ensurethat you enter an e-mail address for each user.

For users authenticated via external means, the User Lookup method must return the user’s e-mailaddress upon login. For some User Lookup methods such as POP3 or IMAP that cannot return thee-mail address, you need to create an account-info script (Section 7.2.5 on page 157) and ensurethat a mail=email-address attribute is always emitted for each login that should be permitted touse locked addresses.

Once all of these steps in Sections 12.2.1 through 12.2.4 have been performed, the Locked Addressfeature is ready to use. Please consult the CanIt-Domain-PRO User’s Guide for details about how touse a Locked Address.


Chapter 13

Attachment Handling

CanIt-Domain-PRO can handle file attachments in a number of different ways. Messages can bedelayed, rejected or held based on the attachment’s type. They can be scanned for viruses and heldor rejected using one or more configured virus scanners. If desired, attachments can also be removedfrom the message and discarded, or held for access via a web-based system.

13.1 General Filename and MIME Type Rules

Whole messages can be rejected or held on a per-stream basis using the Filename Extensions orMIME Types rules. See the section entitled Blocklists, Allow Lists and Rules in the CanIt-Domain-PRO Users Guide for full details.

13.2 Delaying Attachments

On a realm-wide basis, it is sometimes useful to delay certain attachment types temporarily, withoutplacing them in a stream’s quarantine. By delaying these attachments for a short period of time, youcan give your virus scanners and RBLs time to catch up with new virus and spam content.

13.2.1 Configuring the Time Delay


Configure how long CanIt-Domain-PRO should hold attachments by modifying Time in hours todelay messages with Delayed Attachments under Global Settings.

13.2.2 Creating Delay Rules

To create a delay rule, click on Administration and then Delayed Attachments. The Delayed At-tachments screen appears:


188 CHAPTER 13. ATTACHMENT HANDLING

Figure 13.1: Delayed Attachments

To add a rule:

1. Enter a filename pattern in the Filename Pattern box. A filename pattern is normally inter-preted as a filename extension. For example, exe will match a file with the extension .exe.Note that the pattern should not contain a period. If a filename pattern begins with ˆ, then itmatches an entire filename. For example, the pattern ˆbad.exe matches (only) the filenamebad.exe.

2. Enter a comment in the Comment box. This will help you remember why you are delaying thegiven filename pattern

3. Click Submit Changes to add the rule.

Note: Attachment-delaying is global. It cannot be adjusted on a per-stream basis.

13.2.3 How It Works

As an administrator, you may configure any number of file extensions or full filenames to be delayed.When a message arrives matching that filename or extension, it will be held in a special @@DELAYEDstream for the number of hours specified in the Time in hours to delay messages with DelayedAttachments configuration.

Once that time has elapsed, the message is automatically released from the @@DELAYED quarantine,proceeding through the CanIt-Domain-PRO filtering process where normal scanning will proceed asif that mail had just arrived.

Should it be necessary for a message to be released from @@DELAYED early, the admin user (or otheruser with appropriate permissions) may manually release it. Note, however, that a message releasedfrom @@DELAYED may be re-quarantined in its normal stream because of spam-scoring rules. That isbecause messages released from @@DELAYED are scanned by CanIt-Domain-PRO as if they had neverbeen seen before; CanIt-Domain-PRO does not correlate what it believes to be a brand new messagewith anything in the @@DELAYED stream.


13.3. STRIPPING ATTACHMENTS 189

13.3 Stripping Attachments

In addition to delaying, holding or rejecting mail based on characteristics of attachments, CanIt-Domain-PRO can strip attachments out of messages before forwarding the message. You can con-figure CanIt-Domain-PRO to strip out attachments and store them for retrieval via the Web interface,or simply to strip them out and discard them.

Attachment-stripping rules can be set per-stream, but only the realm administrator can create or editattachment-stripping rules; normal users cannot.

To create attachment-stripping rules:

1. Click on Rules and then Attachment Stripping. You see the Attachment Stripping Screen:

Figure 13.2: Attachment-Stripping Rules

2. Enter a filename pattern in the Filename Pattern box. This pattern is interpreted exactly as forDelayed Attachments.

3. Enter a comment in the Comment box.

4. Choose an Action setting to determine how CanIt-Domain-PRO handles the filename pattern:

• Keep in Message indicates that CanIt-Domain-PRO should not strip the attachment out.This setting can be used in a particular stream to override settings in default.

• Strip and Store on Server indicates that CanIt-Domain-PRO should remove the attach-ment and store it in the PostgreSQL database. CanIt-Domain-PRO will also add a messageindicating that the attachment was stripped, and provide a link whereby the message re-cipient can retrieve the attachment.

• Strip and Discard indicates that CanIt-Domain-PRO should remove and discard the at-tachment. CanIt-Domain-PRO will add a note to the message indicating that the attach-ment was discarded and cannot be retrieved.

5. If you chose Strip and Store on Server as the Action, then enabling the Require Approval?checkbox will force administrators to approve the release of held attachments.

6. Click Submit Changes to create the rule.


190 CHAPTER 13. ATTACHMENT HANDLING

13.3.1 Approving the Release of Stripped Attachments

If an attachment rule specifies Require Approval, then when an end-user clicks on the link to retrievethe attachment, he or she will receive a notification stating that an administrator must approve therelease of the attachment, as well as a code to supply to the administrator. To approve the release ofan attachment:

1. Click on Rules : Attachment Stripping

2. Click on the Approve Attachment for Release link near the bottom of the page.

3. Enter the code supplied to you by the administrator.

4. Preview the attachment if necessary.

5. Click on Approve for Retrieval if you wish to allow the end-user to download the attachment.


Chapter 14

URL Proxying

CanIt-Domain-PRO’s URL Proxying feature can help mitigate phishing attacks that trick users intovisiting hostile web sites and entering sensitive information. It does this by rewriting URLs in messagebodies to go to a proxy page that warns users not to enter sensitive information. Users can then clickon a link in the proxy page to visit the original URL. We call the rewriting of the link redirecting thelink.

Here is a screenshot showing what happens when a user clicks on a redirected link:

Figure 14.1: Redirected Link

In Figure 14.1, the original sender sent an email containing the linkhttp://www.cnn.com/WORLD/?hpt=sitenav. CanIt-Domain-PRO redirected the linkto its proxy page. The proxy page shows the user the original link, the server hostname, and (if it canbe determined) the approximate location of the server. It also displays a warning not to enter sensitiveinformation. This can help to educate users about the legitimacy of the site and to remind them not toenter sensitive information.

To continue to the original site, the user merely needs to click on “I understand and wish to follow thelink.”


192 CHAPTER 14. URL PROXYING

14.1 Configuring URL Proxying

By default, CanIt-Domain-PRO proxies only URLs on the Known Phishing URLs list (Section 6.4.2).To enable URL proxying for other URLs, you need to create URL proxying rules.

There are two basic strategies for using the URL proxying feature:

1. Enter a list of safe domains that should not be redirected, and have CanIt-Domain-PRO redirecteverything else. This is the safest approach, but can be annoying as most URLs end up gettingredirected.

2. Enter a list of questionable domains that should be redirected, and do not redirect anything else.If you have a list of commonly-abused domains such as free form-creation sites, this can be aviable method of cutting down on phishing while keeping the annoyance factor to a minimum.

To create URL proxying rules, click on Rules and then URL Proxy. The URL Proxy Rules pageappears:

Figure 14.2: URL Proxy Rules

• To enable URL proxying, set “Enable URL Proxy?” to Yes. This is a normal stream setting, soif you set it in the default stream, it is inherited by other streams in the current realm and allsubrealms.

Note: If the URL proxy is enabled, then CanIt-Domain-PRO proxies URLs for inbound mail anddeproxies them (ie, undoes the wrapping) for outbound mail. An outbound message is definedas one that was forced into a stream by virtue of a Known Networks entry; an inbound messageis one that was not.


14.2. PROXYING KNOWN PHISHING URLS 193

Scanning messages for URLs and replacing them may be expensive, so if a stream does notneed URL proxying, it is best to leave the setting at No. If the setting is No, then any URLProxy Rules are ignored and the Known Phishing URLs list is not used.

• Set your default policy by entering a domain of * and either enabling or disabling the Redirectcheckbox. In Figure 14.2, we proxy URLs by default. The possible policies are:

– No — do not proxy URLs within this domain.

– Yes — always proxy URLs within this domain.

– Only if tagged as spam — proxy URLs within this domain only if the email is tagged asspam and the stream is in tag-only mode.

• Set policies for specific domains by entering them in the Domain box and enabling or dis-abling the Redirect checkbox. Note that a rule for a domain like example.com applies toexample.com and all subdomains unless there is a more specific rule.

URL proxy rules follow the normal stream inheritance. CanIt-Domain-PRO uses the first match-ing rule in the most-specific stream to determine whether or not to proxy a URL.

• You can delete a URL Proxy Rule by enabling the appropriate checkbox in the Delete? column.

• Click Submit Changes to apply your changes.

14.2 Proxying Known Phishing URLs

CanIt-Domain-PRO maintains an updated list of URLs believed to have been used in a phishing at-tempt. If one of those URLs is encountered in a stream where URL proxying is enabled, the URL isalways proxied regardless of any domain rules. In addition, if a user clicks on the modified link, he orshe is not given an option to visit the original URL. Instead, CanIt-Domain-PRO displays a messageindicating that the original link led to a suspected phishing site.

14.2.1 Known Phishing Test Point

The nonexistent URL canit-url-proxy-testpoint.example.com may beused to test the URL proxy. If you send yourself an email containing the texthttp://canit-url-proxy-testpoint.example.com, then CanIt-Domain-PRO shouldtreat it as a known phishing URL.


194 CHAPTER 14. URL PROXYING


Chapter 15

SMTP Server Testing

CanIt-Domain-PRO permits you to run a test SMTP session against a back-end SMTP server. Itdisplays the complete SMTP session and this lets you diagnose problems that may exist.

15.1 An SMTP Primer

Internet email is delivered using a protocol called the Simple Mail Transfer Protocol, or SMTP. SMTPruns over TCP, usually on port 25.

In an SMTP session, there are two computers involved. The machine attempting to send email is theone that initiates the connection, and it is called the SMTP Client. The machine that is intended toreceive the email accepts an incoming connection from the client and is called the SMTP Server.

The data exchanged between an SMTP client and an SMTP server is human-readable plain text. Itconsists of a number of client commands, each of which is responded to with a server reply. The onlyexception is that immediately upon the client connecting to the server, the SMTP server issues a serverreply called the server banner, without waiting for a command from the client. The flow of an SMTPsession is shown in Figure 15.1:


196 CHAPTER 15. SMTP SERVER TESTING

SMTP Client SMTP Server4. Server sends reply

3. Client sends command

SMTP Client SMTP Server

5. Client sends command

6. Server sends reply

SMTP Client SMTP Server

n−1. Client sends QUIT

n. Server closes connection

SMTP Client SMTP Server2. Server sends banner

1. Client connects

Figure 15.1: SMTP Session

Each server reply consists of a three-digit reply code followed by additional text. The first digit of thereply code indicates the success or failure of the preceding client command; the first-digit responsesare as follows:

• 2 indicates a successful reply. It means that the preceding client command succeeded and theserver is waiting for the next command.

• 3 indicates a provisionally successful reply. It means that the preceding client command suc-ceeded, but more information is needed before an overall success or failure status can be re-turned. This reply code is not frequently used and will not appear in CanIt-Domain-PRO’sSMTP tester.

• 4 indicates a temporarily unsuccessful reply, often called a tempfail. It means that the precedingclient command failed, but that it may succeed at some point in the future if the client retriesthe SMTP session. Examples of conditions that could elicit such a response are full disks orproblems reaching a directory server.

• 5 indicates a permanently unsuccessful reply, often called a permfail. It means that the precedingclient command failed and that there is no point in the client retrying later on because it is not


15.2. TESTING AN SMTP SERVER 197

likely to ever succeed. A condition that could elicit such a response is an attempt to send mailto a nonexistent recipient.

15.2 Testing an SMTP Server

The SMTP server-testing feature can be accessed from three places in the Web interface:

• From the Test link next to each Verification Server entry.

• From the Test link next to each Domain Routing entry.

• From the Test links on the Domain Overview page.

When you access the SMTP server-testing feature, the SMTP Server Test Parameters page appears:

Figure 15.2: SMTP Server Test Parameters

To run the test, enter the following parameters:

• Enter the name of the domain to test in the domain name box. Note that this may already befilled in for you.

• Provide the first part of a valid email address. For example, if you are testing the domainexample.com and you know that [email protected] is a valid email address, enterinfo in the second box.

• Optionally enter the server name and IP in the next box. If you leave this box blank, the serverwill be taken from the Verification Server or Domain Routing entry. Note: Only the site admin-istrator can test arbitrary servers. Realm administrators can only test servers that are VerificationServer or Domain Routing entries.

• If a domain has both a Verification Server and a Domain Routing entry, select which server totest. This choice appears only for domains that do in fact have both types of entries.

• Click Run the Test to test the SMTP server.



15.3 SMTP Test Results

Once you run a test, the Test Results page appears:

Figure 15.3: SMTP Server Test Results

The results are displayed in a three-column table. The columns are:

• Time (s) is the time in seconds that has elapsed since the initial SMTP connection was made.

• Source is the source of the message. It is one of Info, meaning an informational messageand not part of the SMTP session; Server, meaning a server reply, or Client, meaning a clientcommand.

• Message is the specific message, reply or command. The server reply codes are highlighted;client commands are shown in bold.


15.3. SMTP TEST RESULTS 199

Let’s step through the SMTP session in Figure 15.3:

1. At time 0.0, CanIt-Domain-PRO successfully connected to the SMTP servervanadium.roaringpenguin.com.

2. At time 0.099, the server replied with a successful response code 220 and its banner.

3. Next at time 0.099, the client sent its first command:EHLO colo3.roaringpenguin.com

4. At time 0.135, the server sent back a multi-line reply. Note that all but the last line have a dashinstead of a space after the reply code. The multi-line reply has reply code 250, indicating thatthe EHLO command was successful.

5. Next at time 0.135, the client sent a MAIL From: command.

6. At time 0.167, the server responded to the MAIL From: command with a successful replycode.

7. Next at time 0.167, CanIt-Domain-PRO informed us that it was going to attempt to send mailto a valid email address.

8. And next at time 0.167, the client sent a RCPT To: command, specifying the email recipient.

9. At time 0.201, the server replied with the code 250, indicating that the preceding RCPT To:command was successful.

10. Next at time 0.201, CanIt-Domain-PRO informed us that it was going to attempt to send mailto a (likely) invalid address.

11. Finally at time 0.201, CanIt-Domain-PRO sent a RCPT To: command that specified a recipi-ent that is very unlikely to exist.

12. At time 0.233, the server replied with the code 550, which indicates a permanent failure. Thepreceding RCPT To: command failed and is not likely to succeed in future.

13. At time 0.234, the client sent a RSET command which throws away everything done so far inthe SMTP session.

14. At time 0.265, the server indicated the success of the RSET command.

15. Next at time 0.265, the client sent a QUIT command.

16. Finally, at time 0.296, the server acknowledged the success of the QUIT command and closedthe connection.


Chapter 16

CanIt Storage Manager

16.1 Storage Manager Concepts

Normally, CanIt-Domain-PRO stores all incident-related data in the PostgreSQL database. For manysites, this works very well and there is no need for any alternate storage mechanisms. However, forlarge sites, storing large amounts of text in the database can be very burdensome, leading to very largedatabases and the consequent very long database dump and VACUUM processes.

To alleviate this problem, CanIt-Domain-PRO ships with a program calledcanit-storage-manager. This program allows you to store large textual data in the filesystem rather than in the PostgreSQL database. The benefits of using the storage manager are:

1. The large amounts of text do not have to be dumped with each database backup, and they do nothave to be VACUUMed.

2. Because the data are stored as ordinary files, you can easily back up and synchronize the datato other machines.

3. canit-storage-manager is optimized for the quick storage and retrieval of textual data,so it reduces the burden on the database server.

4. canit-storage-manager can be run on a different machine from the database server,which improves scalability.


202 CHAPTER 16. CANIT STORAGE MANAGER

16.1.1 Principles of Operation

Figure 16.1 illustrates how the storage manager works:

Web UI

scanner

scanner

canit−storage−manager

SystemFile

ticker

Dis

k tra

ffic

TCP tra

ffic

TCP tra

ffic

TCP traffic

TCP traffic

Figure 16.1: CanIt Storage Manager

• The storage manager daemon runs on one machine and stores data locally on that machine’s filesystem.

• The scanners, ticker and Web interface processes (running on the same machine or in generalon different machines) communicate with the storage manager daemon via a TCP connection.

• The scanners, ticker and Web interface make requests to fetch and store data and the storagemanager daemon carries out those requests.

• Old data are expired by the cron job. The storage manager daemon supports a special “purge”request to delete old data.


16.2. CONFIGURING THE STORAGE MANAGER 203

16.2 Configuring the Storage Manager

Before configuring the storage manager, you need to make the following decisions:

• You need to pick one or more machines to run the storage manager. These machines should befast with plenty of memory and (most importantly) fast disks.

• You need to pick a directory under which the storage manager can store data. (It has to be thesame directory on each machine that runs storage manager.) Be sure there is sufficient diskspace for your expected mail storage! The required disk storage is given approximately by thefollowing formula. (Note that this is a worst-case estimate. It assumes that 100% of your mailvolume is spam and that every message is larger than 8kB and is held locally.)

S = (Dsig×M×V )+(Ddata×8kB×V )+(Ddata×M×V )

where:

– S is the required amount of disk space.

– V is the average number of messages received in a day.

– M is the average size of a message.

– Dsig is the number of days before you expire old Bayes signatures.

– Ddata is the number of days before you expire old data.

For example, if you receive 50,000 messages per day averaging 20kB per message, you retainBayes signatures for 3 days and you expire old data after 28 days, the required disk space is:

S = (3×20×50000)+(28×8×50000)+(28×20×50000) = 42200000kB or about 42GB.

16.2.1 Enabling the Storage Manager

Before using the storage manager, ensure that all machines in your CanIt-Domain-PRO cluster canconnect to the storage manager daemon on port 6568 (or whatever port you choose for it to listen on.)

16.2.2 The Configuration Wizard

Once you have decided on the machine and directory, you can begin configuring the storage managerfrom the Web interface. Click on Setup and then Storage Manager Wizard.

1. First, you are asked whether or not you wish to use the storage manager. Answer Yes. Thenclick Next. The storage manager configuration page appears:



Figure 16.2: Storage Manager Configuration

2. Enter the following information into the wizard:

(a) For each host in your cluster, select whether you want the host to run storage manager inRead/Write mode, Read-Only mode, or not at all. (Normally, you should never run storagemanager in Read-Only mode; this mode is intended only when you are retiring a storagemanager node and want to leave it in the pool until all data on it expires.)

Note: If you change a Storage Manager node from Read/Write to Read-Only or vice-versa, youmust run /etc/init.d/canit-system restart-gracefully on that node af-ter finishing the Storage Manager wizard. Otherwise, the change will not be picked up bythe Storage Manager daemon.

(b) If you have more than one host running a storage manager daemon and you want CanIt-Domain-PRO to store data only on some subset of them, enter the number of hosts onwhich to attempt writes in the “Number of Copies to Write” box.

(c) If CanIt-Domain-PRO is writing more than one copy of the data and you want it to continueoperating even if some writes fail, enter the number of writes required to succeed in the“Success Threshold” box.

(d) Enter the port on which the storage manager daemon should listen. The default port is6568. (The port must be the same for all storage manager hosts.)

(e) If you want to restrict the daemon to listen on a particular IP address, enter it. Normally,you should leave this field blank. If you are running storage manager on more than onehost, you must leave this field blank.

Once you have entered the settings, click Next.

3. Review the settings and then click Finish.


16.2. CONFIGURING THE STORAGE MANAGER 205

16.2.3 Local Configuration

On each host, a number of settings in the [storagemanager] section of /usr/share/canit/canit.conf control various aspects of the storage manager. If you want to change the settings,create a [storagemanager] section in /etc/mail/canit/canit.conf; do not edit /usr/share/canit/canit.conf directly. The settings are:

pidfile (string) A file used by the Storage Manager server to write its process ID andto lock against concurrent Storage Managers. The default value is /var/run/canit-storage-manager.pid.

rootdir (string) The root directory under which data are stored. The default value is /var/lib/canit-storage-manager.

user (string) The UNIX user as which the Storage Manager server should run. The default value isdefang.

client retry delay (integer) specifies the delay in reconnecting to a dead storage manager node. Ifa CanIt-Domain-PRO cluster node fails to connect to a storage manager node, it will not retrythe connection for client retry delay seconds. This can help prevent a dead storagemanager node from bogging down the clients in blocked connect calls.

client connect timeout (integer) specifies the timeout in seconds for a connection attempt to a Stor-age Manager node. The default is 5 seconds.

client operation timeout (integer) specifies the timeout in seconds for a read or write operation to aStorage Manager node once connection has been established. The default is 20 seconds.

order (string) specifies the order in which to try Storage Manager nodes. The default is “auto”,in which case CanIt-Domain-PRO periodically measures the latency to each Storage Managernode and accesses them in order of increasing latency (fastest to slowest). If you want to specifya particular order, set the value to a space-separated list of fully-qualified host names. The hostswill be tried in the order given. If you do not specify all the hosts, then any remaining hosts aretried after the ones specified by the order parameter.

16.2.4 Starting the Storage Manager

Once the settings have been saved, you should log in to each host that will run the storage managerdaemon. Become root and start the storage manager daemon:

# /etc/init.d/canit-system check

(Your canit-system program may be located in /usr/share/canit/scripts/canit-systeminstead.)

The canit-system startup script should run on bootup; it will start the Storage Manager if required.



16.2.5 Data Stored in the Storage Manager

Once the storage manager is enabled, CanIt-Domain-PRO stores the following data in it rather than inthe PostgreSQL database:

• Bayes signatures.

• Message previews (the first portion of an incident’s message).

• Entire messages (if the message is being held locally for some reason.)

In addition, CanIt-Domain-PRO uses the storage manager rather than the database for collecting statis-tics. These statistics are periodically summarized out of storage manager and the summaries are placedin the database.

16.3 Backup Considerations

Once you start using the storage manager, the nightly database dump will not contain all of the infor-mation about incidents. In addition to backing up the nightly database dump, you should also backup the entire storage manager directory tree. (This directory is specified in /etc/mail/canit/canit.conf as the rootdir setting in the storagemanager section. If there is no rootdirsetting, then the default path is /var/lib/canit-storage-manager.)

The files in that directory are ordinary files; you can back them up with tar or rsync or yourfavourite backup tool. However, there are many, many small files within many, many directories andsubdirectories. Test to confirm that your backup tool can handle the directory.

The best time to back up Storage Manager is after the nightly cron job has finished. This is because(a) expired data will have just been purged; and (b) the system should be less busy, resulting in lesscontention for disk I/O.

If you have more than one CanIt-Domain-PRO server (in other words, a cluster) then it is best torun storage manager on multiple CanIt-Domain-PRO servers rather than using a backup tool. Seesection 16.4.

16.4 Running multiple Storage Managers

If you have more than one CanIt-Domain-PRO server running in a cluster then we strongly recommendrunning storage manager on at least two servers. There are several advantages:

• Storage Manager automatically load balances between its nodes;

• Multiple redundant copies of the data eliminate the need for backups;

• When migrating a server, you can skip the process of migrating storage manager as the othernodes will carry the data.


16.5. PS OUTPUT 207

16.5 ps Output

If possible, canit-storage-manager changes the string shown by the ps command to reflectwhat it is doing. For example, ps might show the following output:

canit-storage-manager: 10.0.0.1 scanner_6448 store bayes_sig 19819

The output above means that this instance of the storage manager is connected to the scanner withprocess-ID 6448 on the machine 10.0.0.1. It is currently executing the command “store bayes sig19819”.


Chapter 17

Searching Logs

17.1 Introduction

CanIt-Domain-PRO has the ability to index mail logs in the PostgreSQL database and search them.This can be used to diagnose many mail problems such as missing messages, duplicate messages, etc.

Note: The log-searching feature is available only on our Debian-based appliance build. It is not availablein the source or RPM versions of CanIt-Domain-PRO. See the CanIt-Domain-PRO Installation Guidefor details on installing the log-searcher.

In addition to presenting search results from the log files, CanIt-Domain-PRO also annotates the log-lines to provide a clear explanation of what each line means. This can greatly ease troubleshooting.

17.2 Log Basics

CanIt-Domain-PRO uses the Sendmail program to transfer mail. It also uses the MIMEDefang filter-ing tool as the basis for its filtering. There are therefore three sources of log lines:

1. Sendmail.

2. The core MIMEDefang tool.

3. CanIt-Domain-PRO itself.

The log indexer groups log lines for a given message into a log document. A log document consists ofthe set of log lines that describe the process of one message transmission through the CanIt-Domain-PRO system.

The common element between different log lines that allows them to be grouped together is the Send-mail queue ID. This is an identifier assigned by Sendmail to each message transmission. A typicalqueue ID might look like this: oBGIkIUj026238


210 CHAPTER 17. SEARCHING LOGS

17.3 Searching the Logs

There is a 30-minute delay between a log-line being created and the indexer indexing it. Therefore,you can search for log lines starting as far back as your logs go up until 30 minutes before the currenttime.

17.3.1 Performing a Search

Note: Only the system administrator or realm administrators can use the log-searching facility. In addition,the user must have permission to see quarantine contents.

To search the logs, click on Administration : Search Logs. The Log Search page appears:

Figure 17.1: Log Search Page

The Log Search page lets you build up a complex search query and then execute it. Here’s howlog-search queries work:

• Start date and End date restrict the time interval over which the search is performed. These


17.3. SEARCHING THE LOGS 211

are not actually part of the query.

• A query is a list of zero or more groups. Each group is evaluated as a unit before evaluating thenext group.

• Each group consists of one or more expressions. Each expression is evaluated as a unit.

• An expression consists of a field, a relation and some data. These will all be explained soon.

• Within a group, expressions are joined with AND, OR, AND NOT or OR NOT. The ANDoperator is evaluated with higher precedence than OR. (If you include NOT, the NOT negatesthe next expression.) Thus, for example, a query like:

(X = 1) AND (Y = 2) OR (A = 3) AND NOT (B = 4)

is evaluated as:

((X = 1) AND (Y = 2)) OR ((A = 3) AND (NOT (B = 4)))

• Within a query, groups are joined with AND, OR, AND NOT or OR NOT. Again, the ANDoperators have higher precedence than OR.

17.3.2 Fields

The possible fields for searching logs are:

• Incident ID lets you search for a specific CanIt-Domain-PRO incident ID.

• Queue ID lets you search for a specific Sendmail Queue ID.

• Sender lets you specify an envelope sender’s email address.

• From: Header Address lets you specify the email address appearing in a message’s From:header.

• List of Rules Hit lets you search the tests=xxx field of CanIt-Domain-PRO’s what= logline. The most useful way to use this field is with the contains relation. If you use thatrelation, you can search various rule types as follows:

– To search for a SpamAssassin rule such as HTML MESSAGE, enter the rule identifier ex-actly as shown.

– To search for a Custom Rule with ID N, search for: ;CN( where the semicolon, the C andthe ( are literal.

– To search for an SPF result of xxx, search for: SPF(xxx:

– To search for an DKIM result of xxx, search for: DKIM(xxx:

– To search for a DMARC result of DMARC POLICY xxx, search for:DMARC(DMARC POLICY xxx).



You can also search for filename extensions using List of Rules Hit. For example, to search fora docx extension, use:

List Of Rules Hit Contains ext:docx

To look for a zip-within-a-zip, use:

List Of Rules Hit Contains ext:>zip

• Recipient lets you specify an envelope recipient’s email address.

• Source Relay IP lets you restrict results to messages relayed from a specific IP address.

• Destination Relay IP lets you restrict results to messages relayed to a specific IP address.

• Subject lets you specify the subject of a message.

• Message ID lets you specify a Message ID (found in the Message-Id: header of an email.)

• Reporting Host lets you restrict the search to messages processed by a particular host. Notethat you need to specify the host name as it appears in the log file.

• Classification lets you restrict messages based on their classification. Possible values for clas-sification are:

– Accepted

– Rejected

– Discarded

– Greylisted

– Pending

– Tagged

– Streamed

• Stream lets you restrict results to messages within a given stream.

• Score restricts the results based on score.

• Reason restricts results to messages with the given reason=xyz entry in their logs.

• Detail restricts results to messages with the given detail=xyz entry in their logs. For moreinformation about the reason and detail fields, see Appendix H, “Logging”.

• Realm lets you restrict messages to a particular realm. The Realm field is displayed only if youhave access to more than one realm.


17.4. SAVING LOG SEARCHES 213

17.3.3 Creating a Log Search Query

To create a log search query:

• Starting with a blank query, select a field and relation for the search. Enter the data to search forand click Add

• Continue to refine the query by selecting additional fields and relations and entering additionaldata. Also, select one of AND, OR, AND NOT or OR NOT as the logical operator to join thenew expression to the existing query.

• Click Add to add the new expression to the current group, or click Add as New Group to starta new group.

• Click Delete to delete the most recently-added expression if you made a mistake.

17.4 Saving Log Searches

CanIt-Domain-PRO permits you to save a log search and call it up later to redo the search.

To save a log search:

1. Create the log query in the normal manner.

2. Enter the name under which you would like to save the search in the box to the right of the SaveSearch As... button.

3. Click Save Search As...

17.4.1 Managing Saved Log Searches

To manage saved log searches, click on Managed Saved Searches. The Saved Log Searches pageappears:

Figure 17.2: Saved Log Searches

To recall a saved log search, click on the name of the search. The log-search page will appear with thequery loaded from the saved search.



To add a comment to a saved log search, enter the comment in the appropriate box and click SubmitChanges.

To delete saved log searches, enable the appropriate checkboxes in the Delete? column and clickSubmit Changes.

17.5 Log Search Results

After you click Add and Search to submit a log search request, CanIt-Domain-PRO returns a list ofmatching results. This list might look something like Figure 17.3:

Figure 17.3: Log Search Results

Within the results page:

• Click on the small up- or down-arrows next to each column to sort by that column in ascendingor descending order. The current sort order is shown by the red arrow.

• Click on a Queue ID link to view the detailed log lines for that queue ID.

• If there is an incident associated with the logs, the message subject will be a link. Click on it tosee the Incident Details page.

Note: Sometimes a group of log lines does not contain complete details about a message. In this case,CanIt-Domain-PRO acts as follows:

• If the subject could not be determined, CanIt-Domain-PRO displays the subject as (NotLogged).

• If the stream could not be determined, CanIt-Domain-PRO assumes the default stream.

• If the realm could not be determined, CanIt-Domain-PRO assumes the base realm.

It is important to remember that for queue retries and other fragmentary groups of logs, the subject,realm and stream may not be able to be determined.


17.5. LOG SEARCH RESULTS 215

17.5.1 Detailed Results

If you click on a queue ID, the Detailed Results page appears:

Figure 17.4: Log Search Details

This shows each log line related to the message transmission. To see the timestamp in a more readableformat, hover the mouse cursor over the timestamp. For a detailed explanation of a log line, click onthe question-mark icon next to the line. You can expose details for all log lines by clicking Show AllExplanations.

Finally, if you need the raw log lines (for example, to send to someone for analysis), click on ShowRaw Logs.

17.5.2 Downloading Log Lines

At the bottom of the log results page, you will see one or two links:

• Bookmarkable Link is a link that you can copy and paste or send via email to redo thecurrently-displayed log search.

• Download Logs is a link that permits you to download all logs that correspond to a particularquery. The downloaded logs are in plain-text format that can be opened with a text editor.

CanIt-Domain-PRO does not always provide a Download Logs link. If the number of log searchresults is greater than the internally-configured MaxDownloadableLogs setting (default 100 log en-tries), then CanIt-Domain-PRO does not permit logs to be downloaded.

The CanIt-Domain-PRO site administrator can increase the limit by creating a file under the CanIt-Domain-PRO web tree called site/config.d/99 logentries.php with the following con-tent:

<?phpglobal $Config;



$Config['MaxDownloadableLogs'] = 500;?>

In the previous example, the limit was raised from 100 to 500.

When you download log lines, they are grouped by log host. Within a given log host, the lines aresorted chronologically. To sort all lines chronologically regardless of log host, use your text editor’sline-sorting feature or a utility similar to the UNIX sort command.

17.6 Forwarding Logs

CanIt-Domain-PRO has the ability to forward logs on a per-realm basis to other machines using thesyslog protocol.

17.6.1 Enabling Log-Forwarding

By default, CanIt-Domain-PRO will not forward logs. To enable log-forwarding, the CanIt-Domain-PRO site administrator must edit the file /etc/mail/canit/canit.conf on each CanIt-Domain-PRO log host and add the following lines:

[logindexer]forward logs = yes

17.6.2 Configuring Log-Forwarding

To configure log-forwarding, click on Administration : Forward Logs. The Log Forwarding Pageappears:

Figure 17.5: Log Forwarding Page

Note: Only the CanIt-Domain-PRO site administrator can configure log-forwarding for arbitrary realms. Ifyou are a realm administrator, the Log Forwarding Page allows you to configure log forwarding onlyfor your current realm.


17.6. FORWARDING LOGS 217

To forward logs for a particular realm:

1. Enter or select the realm name in the Realm column.

2. Type the IP address or host name of the destination host in the Log Host column. If you useUDP transport, you can enter multiple log hosts in a comma-separated list; in this case, loglines will be forwarded to each host. Additionally, you can use a different port for each host byfollowing the host name or IP address with /port.

If you use TCP transport, then you can only enter a single log host and cannot override the port.

3. Enter the port number in the Port column. The standard SYSLOG port is 514.

4. Select the transport (either UDP or TCP) from the Transport column.


To disable forwarding for a realm, delete the entry with the Delete? check box, or enter a blank stringfor the host name.

Note: Forwarded logs are always forwarded with the mail facility and info priority, regardless of theoriginal priority. Also, the entire original log line is forwarded including a high-resolution time-stamp.The receiving machine may log some redundant information with each received log line because ofthe way it is forwarded.

Because CanIt-Domain-PRO must correlate log lines and ensure that all lines pertaining to a realmare forwarded (and no lines not pertaining to the realm are inappropriately forwarded), logs are notforwarded in real-time. There may be a delay of up to 30 minutes between a line being logged on theCanIt-Domain-PRO system and the line being forwarded to the remote host. Nevertheless, the originaltimestamp is preserved.


Chapter 18

Tips

Managing spam requires constant attention, but there are many things you can do to reduce the work-load of the administrator. This chapter offers advice for fine-tuning CanIt-Domain-PRO and makingit more effective.

18.1 Greylisting


In the past, spammers would use open SMTP relays to send spam. With the advent of inexpensiveresidential broadband, many spammers use special software to send bulk mail directly from their PC’s.

Because spammers want wide distribution, they want each message to be sent as cheaply as possible.Some spam software, therefore, ignores SMTP errors if a message cannot be delivered.

CanIt-Domain-PRO can deal very effectively with software that never retries by sending a temporaryfailure indication at the end of DATA when mail from an unknown sender arrives. If you set the“Tempfail unknown senders on first transmission” stream setting to Yes, then CanIt-Domain-PROuses the combination of sender e-mail address, recipient e-mail address, sending relay IP addressand message subject to calculate a hash. If this hash has never been seen before, CanIt-Domain-PRO tempfails the message. Once the hash reappears, CanIt-Domain-PRO marks the host as “knownto retry” and lets the message to proceed to content-scanning. A host marked “known to retry” isallowed to bypass greylisting for 40 days.

There are some down-sides to using greylisting. Valid mail from new senders may be delayed byanywhere from 15 minutes to four hours, depending on the retry interval on the sending relay. Youcan avoid this delay by setting up a secondary MX record. In fact, you can simply give the CanIt-Domain-PRO machine a virtual interface with another IP address and publish this other IP address asa secondary MX record. In this way, when proper SMTP relays receive a temporary failure indicationon the primary MX machine, they immediately try to send to the secondary MX machine. Often,spamware won’t retry.

On a similar note, CanIt-Domain-PRO will not issue temporary failures for messages relayed fromany server in a Known Network with Skip Greylisting configured (see Section 5.7 on page 65). If


220 CHAPTER 18. TIPS

a message is received by such a server, greylisting will not be used. In some cases, this can causegreylisting statistics to be skewed. For example, if mail is initially received by a CanIt-Domain-PROserver and marked as greylisted, then is received by a secondary MX server and either relayed to theCanIt-Domain-PRO server, or to an internal mail server, the message will appear in the CanIt-Domain-PRO statistics as having been greylisted, even if it was received and processed.

In general, we find that setting Tempfail unknown senders on first transmission to Yes is a cheapand effective way to reduce spam.

WARNING: Some mailing list programs use “disposable” sender addresses which always change.These lists do not work well with greylisting. To work around the problem, you should always allowthe domain of the mailing list sender.

CanIt-Domain-PRO tries to detect disposable-address schemes. It ignores everything in the senderaddress following a plus sign or a dash followed by a digit. These rules catch most common methodsfor generating disposable addresses, but they are not exhaustive.

18.2 Don’t Trust Sender Addresses

Many spammers use one-time disposable sender addresses. Many addresses are not even valid. We donot recommend blocking addresses unless you receive many different spam messages from the sameaddress. Therefore:

Blocking individual addresses is usually not effective. Always allowing known goodaddresses (for example, mailing-list sending addresses) can be very effective. The senderreport may, however, highlight a persistent spam sender address which is worth blocking.

18.3 Don’t Trust Sender Domains

Just as sender addresses are often fake, sender domains are too. However, some domains are knownspammers and these can be profitably blocked. The tip:

Blocking entire domains can be effective under limited circumstances. Always al-lowing domains is generally a bad idea because spammers often fake mail from gooddomains. Holding all mail from free e-mail services like Hotmail and Yahoo can be ef-fective if you use it in conjunction with always allowing of known good senders fromthose services. Use the domain report to help make these decisions.

18.4 You May Trust Relay Hosts

It is rather difficult to fake the IP address of the SMTP relay host, so this attribute can usually betrusted. We recommend using a DNS-based blocklist service in your Sendmail configuration file orthe CanIt-Domain-PRO GUI to reject the most obvious offenders. However, if you receive multiplespam messages from a given relay host, it can be effective to block the host:


18.5. CUSTOM RULES 221

Blocking a repeat-offender relay host is effective. Always allowing known good hostssuch as internal hosts is effective and recommended. Use the host report to determinewhich hosts are persistent spam relays.

18.5 Custom Rules

18.5.1 General Recommendations

There are a few custom rules which are quite effective:

1. If you know that your CanIt-Domain-PRO server only accepts inbound mail from the Internet,then no server should ever claim to be in your domain in the HELO command. If your CanIt-Domain-PRO server is called canit.mydomain.tld, a custom rule to add 5 points if HELOends with mydomain.tld can be very effective. In fact, you might want to make high-scoringrules which automatically reject messages with obviously-fake HELO arguments.

2. Similarly, no machine should ever put an IP address as the argument of HELO. Some spammersuse random IP addresses here to confuse spam-reporting tools. A custom rule which “regexp-matches” HELO against ˆ\d+\.\d+\.\d+\.\d+$ can be quite effective.

3. Custom rules which specify Sender contains “offer”, “bounce”, “return” and “noresponse” canoften trap spam. You should use only moderate scores on these rules, because some legitimatemail comes from such senders. However, adding a rule which scores around 3 for these patternscan help catch a lot of spam which might otherwise sneak under the scoring threshold.

4. Subject-matching rules for the most obnoxious spams are very effective. For example, Sub-ject regexp-match rules against v\Sagra and (increase|enlarge).*penis are veryeffective.

18.5.2 Things to avoid

Be very careful when writing custom rules, especially rules that can match on the message body.For example, a straightforward rule that contains “cum” in the body will match mail containing mailcontaining “document”, “cumulative”, “modicum” and at least 64 other common English words. Sim-ilarly, “sex” will match “sexton”, “Essex” and others.

If you want to match words in a message body, we recommend that you use a regular-expressionmatch, and use Perl’s word-boundary operators. For example, the Perl regular expression \bcum\bmatches the word “cum”, but not “document”, “cumulative” or “modicum”.

18.6 Group High-Scoring Messages Together

We recommend that you set the default sort order to sort by Score, Descending. This groups high-scoring messages at the beginning and low-scoring messages at the end of the pending list. This makesit easier for the spam-control officer to dispose of the messages.


222 CHAPTER 18. TIPS

18.7 Roaring Penguin Best-Practices

At Roaring Penguin Software Inc., we’ve spent quite a bit of time analyzing spam and spammers. Youmay wish to try out some of our anti-spam rules to see if they work well for you. Here is a quicksummary of the rules we use; they may inspire you to develop your own anti-spam rules.

• We use custom rules to add 4 to any message whose Sender contains “offer”, “noresponse”,“remove”, “marketing” or “promo”. These rules may be a touch aggressive for very busy sites,but are quite effective for smaller sites.

• Another custom rule adds 1.2 to any Relay containing “[” (left square bracket.) This indicatesa reverse-DNS failure on the sending host, which is mildly correlated with spamming.

• We use a Spam threshold of 4.6, because we find the default of 5 is somewhat conservative.

• We use a discard threshold of 20; this seems quite safe.

• We set Tempfail unknown senders on first transmission to Yes. Again, this may be unaccept-able for some sites.

18.8 General Anti-Spam Tips

18.8.1 Use Receive-Only Addresses on your Web Site

Spammers love to extract e-mail addresses from Web sites, and not only do they use them for theobvious purpose of spam targeting, but also they use them as fake sender addresses.

Therefore, we recommend a general policy of publishing only generic e-mail addresses on your Website, like [email protected] and [email protected]. When you replyto inquiries, always use a real, personal e-mail address like [email protected]. This hastwo benefits:

1. If someone sends e-mail purporting to come from [email protected], you knowimmediately that it is spam, and you can reject it. You can block all your generic addressesinside CanIt-Domain-PRO.

2. If someone complains about receiving e-mail from one of the generic addresses, you can pointto your policy and assure the recipient that the sender address was faked.

18.8.2 Do Not Reply to Spam

Do not ever reply to spam e-mail; such replies simply serve to validate your e-mail address. Similarly,do not visit Web sites purporting to offer opt-out services; they also serve to validate your address forfurther spamming.


Chapter 19

Security

Running a secure CanIt-Domain-PRO installation is relatively straightforward, but there are manyissues you have to watch out for. This chapter gives you guidance on how to secure your CanIt-Domain-PRO installation.

19.1 Don’t Run as Root

The most basic security principle is to run as little software as root as possible. Therefore:

• Always create the Sendmail smmsp user and group, and do not run Sendmail suid-root. Instead,the permissions on the Sendmail executable should look like this:-r-xr-sr-x root smmsp sendmailThat is, the sendmail binary should be owned by root, group smmsp and have mode 2555.

• Always create the MIMEDefang defang user and group, and run MIMEDefang as defang. In/etc/mail/canit/canit.conf, enable mx user=defang in the [mimedefang]section.

19.2 Ownership and Permissions

All system configuration directories like /etc and their ancestors and descendants should be ownedby root and writeable only by root. Here are suggested ownership and permissions for various filesand directories. Note that where we use group root, your system may use wheel or some other groupfor root-owned files.

File or Directory Owner Group Mode/etc/mail/canit and ancestors root root 0755/etc/mail/canit/canit.conf apache defang 0640/var/spool and ancestors root root 0755/var/spool/MIMEDefang defang defang 0700/var/spool/MD-Bayes defang defang 0755/var/lib/canit-storage-manager defang defang 0700The PHP files in Apache’s Web space root root 0644


224 CHAPTER 19. SECURITY

19.3 SSH

The various nodes in a CanIt-Domain-PRO cluster communicate via SSH. Each node must be able toSSH to all other nodes on port 22.

For intra-cluster communication to work, root SSH login must be permitted. However, you do notneed to permit general root login because the CanIt-Domain-PRO nodes only use a forced commandfor communication. The safest setting in /etc/ssh/sshd config is therefore:

PermitRootLogin forced-commands-only

19.4 PostgreSQL Security

By default, PostgreSQL trusts any connection coming from the local host.

Therefore, if you use PostgreSQL on your CanIt-Domain-PRO server with the default access rules,do not allow normal users to have shell accounts on the CanIt-Domain-PRO server. This cannot beemphasized strongly enough: If you allow normal users shell access on the CanIt-Domain-PRO serverwith PostgreSQL’s default setup, anyone can access or change the spam database.

If you must allow shell accounts on the CanIt-Domain-PRO server, then you must password-protectyour PostgreSQL installation. See the PostgreSQL documentation (“Authentication Methods” section)for details. You must also protect your database passwords:

• The file /etc/mail/canit/canit.conf must be owned by apache and group defang.Both the defang user and the apache user need read-access to these files, which should havemode 0640. (We assume your Web server runs as user apache; if not, substitute the Web serveruser as appropriate.)

For best security, we strongly recommend that you do not allow ordinary users to have shell accountson your mail server. If the CanIt-Domain-PRO database server is on a different machine, you shouldnot permit shell accounts on that machine either.

19.5 PHP Security

PHP has a parameter called register globals, which automatically sets global variables basedon GET, PUT or COOKIE variables. This setting may be a security risk, and CanIt-Domain-PRO doesnot require it. We strongly recommend that you set register globals to off.

19.6 Network Security

When you log on to CanIt-Domain-PRO, your username and password are transmitted in cleartext.While you interact with CanIt-Domain-PRO, your browser passes a session cookie back so CanIt-Domain-PRO can keep track of your session. Both your password and the cookie are vulnerable to


19.7. BACKUPS 225

network sniffing. If you interact with CanIt-Domain-PRO over an untrusted network, or a networkwhose traffic may be sniffed, you should use HTTPS and SSL encryption. Setting this up is beyondthe scope of this manual, but CanIt-Domain-PRO should operate with no changes over HTTPS.

19.7 Backups

The daily CanIt-Domain-PRO cron job dumps a text backup of the spam database to the file/var/spool/Canit-Spam-DB-Backup/SPAM-DATABASE-BACKUP. You should back thisfile up regularly in case the CanIt-Domain-PRO server suffers a hardware or other problem. Youshould also make sure the file is not readable by normal users.

You should also back up the entire directory tree rooted at /var/spool/MD-Bayes. If you areusing the Storage Manager, you should also back up the Storage Manager directory on each StorageManager node.

Some CanIt-Domain-PRO settings are stored in /usr/share/canit as well as/etc/mail/canit; you should back up that directory any time that you change a file in it.You may wish to back up /etc/mail in its entirety to capture Sendmail configuration files in yourbackup as well.

See Section E.3 for more information on automating backups to a remote location.

Note: When restoring from backups, never replace existing /etc/mail/ or /usr/share/canit fileswith backed up versions! Rather, use your backup versions as reference.

Finally, please remember to back up any customizations you have made to your CanIt-Domain-PROinstallation, including web interface files, custom account-info or other scripts, et cetera.

Note: When restoring from backups, be careful when replacing web interface files, especially (but not only)if you are restoring to a different version of CanIt-Domain-PRO than that from which your backupwas made.


226 CHAPTER 19. SECURITY


Appendix A

The Domain Configuration Wizard

A.1 Introduction

The Domain Configuration Wizard provides a simple way to quickly configure the most importantsettings for a domain. All of the pages in the Domain Configuration Wizard are available in greaterdetail in the Setup and Administration menus. However, because the Domain Configuration Wizardcentralizes the important settings in one simple workflow, you may prefer to use it to set up newdomains.

To access the Domain Configuration Wizard, click on Setup and then Wizards. Click on DomainConfiguration Wizard.

A.2 Entering the Domain Name

The first step in the Domain Configuration Wizard requires you to enter a domain name. (Figure A.1).Enter the domain name and click Next.

Figure A.1: Domain Configuration: Enter Domain Name

A.3 Picking a Realm

In the next page (Figure A.2), you are prompted to select a realm name. Enter the realm name andclick Next.


228 APPENDIX A. THE DOMAIN CONFIGURATION WIZARD

Figure A.2: Domain Configuration: Enter Realm Name

You may type the name of an existing realm, in which case CanIt-Domain-PRO maps the new domaininto that realm. Or you may enter a new realm’s name, in which case the realm will be created and thedomain will be mapped into that realm. If no data for the new domain exists yet, CanIt-Domain-PROwill suggest a realm name based on the domain name.

A.4 Configuring Streaming

The next step (Figure A.3) requires you to choose how mail for the domain should be streamed.Streaming is explained in detail in Chapter 4.

Figure A.3: Domain Configuration: Configuring Streaming

You can configure streaming in several ways:


A.5. CONFIGURING AUTHENTICATION 229

• You can simply chop the domain part off the e-mail address so that mail [email protected] goes into the stream user.

• You can chop the local part off the e-mail address so that mail for [email protected] into the stream example.net.

• You can keep the entire e-mail address as the stream name. This is the recommended methodfor most installations.

• You can invoke the User Lookup Wizard to set up a more complex streaming method (forexample, using LDAP). The User Lookup Wizard is described in Chapter 7.

Note that if you have created User Lookup methods (either in the past or after stepping through theUser Lookup Wizard from the Domain Configuration Wizard), you will be presented with additionalchoices for streaming.

A.5 Configuring Authentication

Once streaming has been configured, you will be asked to configure authentication (Figure A.4).

Figure A.4: Domain Configuration: Configuring Authentication



To allow end-users to log into CanIt-Domain-PRO and manage their quarantines, you can set up anauthentication mechanism. From the Domain Configuration Wizard, you have several choices:

• IMAP allows you to authenticate users against an IMAP server.

• POP3 allows authentication against a POP3 server.

• Other allows you to skip setting up authentication. You can do it at a later time, or (if you do notwant to allow end-users to log in) skip it entirely. You can also step through the User LookupWizard to set up a more complex authentication mechanism.

If you select IMAP or POP3, you will be prompted to enter the name (or IP address) of the IMAP orPOP3 server. If CanIt-Domain-PRO should strip the domain name off the login name before attempt-ing to authenticate, set the “Strip domain name from login” parameter to Yes. You can also configureCanIt-Domain-PRO to validate SSL certificates and to use (or require) an encrypted connection to thePOP3 or IMAP server.

If you step through the User Lookup Wizard to create an authentication method, the newly-createdmethod will be presented as an authentication choice when you return to the Domain ConfigurationWizard.

A.6 Configuring Routing and Verification

Finally, CanIt-Domain-PRO will ask you to configure routing and verification (Figure A.5).

Figure A.5: Domain Configuration: Configuring Routing and Verification


A.7. SUMMARY 231

Note: Configuring routing via the Web interface is only available on CanIt-Domain-PRO appliance builds.If you are not running an appliance build, you will need to configure routing using Sendmail’s mail-ertable feature; consult the Sendmail documentation for details.

To route mail for the domain, enter the host name or IP address of the back-end SMTP server that willaccept e-mail for the domain.

We strongly recommend configuring some method for CanIt-Domain-PRO to validate recipient ad-dresses. If you do not validate recipient addresses, CanIt-Domain-PRO is forced to accept mail forany address withing the domain, likely resulting in many failure notifications.

If your back-end mail server validates recipients during the SMTP transaction, enter its name or IPaddress as the verification server. If it does not, you will have to leave the verification server blankand use some other method (such as LDAP streaming) to validate recipients.

A.7 Summary

After configuring routing and verification, CanIt-Domain-PRO will display a summary of your set-tings. Click Finish to make them take effect.


Appendix B

Release Notes

Version 10.2.0 released on 2018-02-13

• NEW FEATURE: Users can select their time zone (it is an inheritable stream setting) and alldates/times in the UI are expressed and accepted in the user’s time zone.

• NEW FEATURE: CanIt features experimental integration with Microsoft’s Azure Active Di-rectory (a hosted directory service.)

• IMPROVEMENT: Greatly improved and simplified the database failover code. It’s now easierto set up and more bulletproof. A lot of useless settings have been removed and the code hasbeen cleaned up considerably.

• IMPROVEMENT: We use the “qpdf” helper program to analyze PDF documents rather thanthe older, slower and buggier “podofopdfinfo”

• IMPROVEMENT: In the case of multiple DKIM signatures, we evaluate them all and add upthe corresponding scores.

• UPDATE: Update ClamAV to 0.99.3.

• BUG FIX: Improved detection of Microsoft Office documents with executable content insidethem.

• BUG FIX (Archiver): Properly MIME-encode subjects when remailing archived messages.


• SECURITY IMPROVEMENT: Our old 1024-bit DSA key for Roaring Penguin service accesshas been replaced with a 4096-bit RSA key.

• POLICY CHANGE (CanIt-Domain-PRO only): Auto-whitelist rules are *never* created in thebase:default stream. Such rules would apply site-wide and we consider auto-creation of site-wide rules to be too dangerous.


234 APPENDIX B. RELEASE NOTES

• IMPROVEMENT: This version includes small improvements to the speed of custom rule eval-uation.

• IMPROVEMENT: The OfficeMacro AutoOpen test also detects MS Office files that attempt toexecute code without using macros.

• CHANGE: If you mark a URL like that has only a hostname such as www.example.com/ as aphishing URL, then then *all* URLs that begin with www.example.com/ will be considered tobe phishing URLs.

• CHANGE (Log Search Only): A configuration setting in config.php allows you to limit theduration of log search queries to avoid having ridiculous searches launch very expensive SQLqueries that run for a long time.

• CHANGE (Archiver only): A search with no realm qualification searches all realms owned bya realm administrator rather than just the current realm.

• BUG FIX: Release 10.1.8 broke the code that checks for verification server loops; this releasefixes it again. In addition, CanIt now raises an anomaly if it detects a verification server loop.

• BUG FIX: canit-setup-appliance on Debian Stretch would not detect any Ethernet interfaces.This is now fixed.

• BUG FIX: Upon upgrade, the Roaring Penguin service SSH key would be enabled even if ithad explicitly been disabled before. Now, when you run canit-service-key –disable, this fact isremembered and the key is never automatically re-enabled.

• BUG FIX: The output of “lsar” changed format on Debian Stretch; we now handle both the oldand new formats correctly.


• IMPROVEMENT: CanIt will attempt to negotiate STARTTLS when it runs against a verificationserver. This only works on Debian Jessie or higher and only if the verification server advertisesSTARTTLS.

• IMPROVEMENT (Archiver Only): In an Archive search, you can use the “is” relation with the“Subject” field.

• IMPROVEMENT: Added support for PostgreSQL 10. At this point, the support for PostgreSQL10 is considered experimental.

• IMPROVEMENT (Log Searching): By default, log search queries for non-site-administratorsare cancelled after 5 minutes. This limit is configurable in site/config.php; see con-fig.d/20 log search.php for details.

• BUG FIX: The URL Proxy code would not rewrite URLs in a released incident even if the rulessay that it should. This has been fixed.


235

• BUG FIX: The default login filters for ActiveDirectory user-lookups now include (!(userAc-countControl:1.2.840.113556.1.4.803:=2)) in the login search filter to prevent disabled AD ac-counts from being able to log into CanIt. Existing user-lookups are NOT changed, so you willhave to add this clause manually if you want to apply it to existing lookups.

• BUG FIX: In certain rare cases, Custom, Archiver, etc. rules could compile to Perl code thatdid not evaluate as intended. This has been fixed.

• BUG FIX: Implemented a workaround for a Linux kernel bug in the nightly cron job, whichwould sometimes complain about not being able to remove directories from a tmpfs file systembecause they were not empty.

• BUG FIX: Fixed a typo in the CanIt::NewDomain module that could result in unnecessarily-high database load on very busy systems.

• BUG FIX (Archiver Only): The search form defaults the relation for “Realm” and “Stream”fields to “is” rather than “contains”.

• BUG FIX (Secure Messaging only): In some cases, activation codes could fail to work whensetting up a new account. This has been fixed.

• BUG FIX (Secure Messaging only): When running on a version of PHP older than 5.6, usersexperience authentication issues when trying to log in to the Secure Messaging portal. This bugwas introduced in 10.1.7 and is fixed in this release.


• NEW FEATURE: Roaring Penguin has a service that tracks when domains are first seen; thiscentral database is now available for all CanIt installations and is used by the NewlySeenDomaintest.

• NEW FEATURE: The ability for users to see inherited rules is now controlled by a permissionrather than always being available.

• NEW FEATURE: A Custom Rule can match against the MX hosts of the envelope sender. Thislets you detect parked domains, for example.

• MAJOR IMPROVEMENT: The URL Proxy landing page follows redirects to show the ultimatedestination of a link. It adds extra warnings if the link appears to point to an executable file oran archive.

• IMPROVEMENT: If the greylist delay is increased due to a DNSBL hit, this fact is logged.

• CHANGE: The “Welcome to CanIt” line shown on the login page has its own CSS class so itcan be hidden by a theme customization.

• CHANGE: The term “slave” with reference to scanning processes has been replaced by the term“worker”.



• BUG FIX: A couple of places where canitd jobs could potentially hang have been fixed byimposing timeouts.

• BUG FIX: If the auto-whitelist flag is set on a Known Network, auto-whitelisting is appliedbased on both the connecting server and the originating server in the cases when they are differ-ent.

• BUG FIX: The various .pid files that CanIt creates are placed in /var/run rather than/var/spool/MIMEDefang. This tightens up the permissions.

• BUG FIXES (Secure Messaging): Incorrect template name for Secure Messaging was fixed.Minor database schema error was fixed; this does not affect functionality, only performance.

• BUG FIX (Secure Messaging): Secure Messaging now works correctly on Debian 9 (“Stretch”)and GnuPG version 2.

• BUG FIX (Secure Messaging): Replies via the web interface would not be encrypted under arare edge-case condition. This has been fixed.


• MAJOR NEW FEATURE: Almost all pages under the Rules top-level menu permit you to seeinherited rules as well as the rules specifically in the stream you are viewing. You can togglethe inherited-rule view on or off. (Some rule types are considered sensitive and only the siteadministrator can view inherited rules in that situation.)

• NEW FEATURE: CanIt tracks all top-level domains from which mail has been received. 30days after upgrading to 10.1.6, it will start adding one point for messages from “newly-seen”domains — that is, domains that first were seen 7 days or less ago. The score is adjustable underRules > Plugins > NewlySeenDomain

• MAJOR IMPROVEMENT: For outbound mail (that is, mail forced into a stream by a KnownNetworks entry), CanIt will undo any URL-proxying that was done on the way in. This requiresthe URL Proxy feature to be enabled in the outbound stream.

• IMPROVEMENT: The GUI for Sender, Domain and Network Rules has been made consistentwith other rule GUIs: A new blank row at the top of the page for a new rule, and checkboxes ina “Delete?” column for deleting rules.

• IMPROVEMENT: In /etc/mail/canit/failover.conf, you can specify NOT to supply the -z flagto rsync with the setting compress rsync=0. If you have a fast link between the primary andhot-standby database servers, it’s better not to compress.

• UPDATE: We now have packages for Debian 9 (“Stretch”) as well as Debian 9 ISO images.The Debian 9 packages and images are currently considered experimental.

• UPDATE: Several new country codes were added to Rules > Countries.

• UPDATE: CanIt’s public suffix list has been updated.


237

• MINOR FIX: All CanIt links referring to http://www.roaringpenguin.com have beenchanged to refer to https://www.roaringpenguin.com instead.

• MINOR FIX: If a Verification Server tempfails mail because of an unresolvable domain, we nolonger raise an anomaly.

• MINOR FIX: The nightly system checker handles domains with non-resolving NS records bet-ter than before; it raises a descriptive anomaly rather than issuing mysterious error messages tothe logs.

• BUG FIX: The Custom Rule Compiler failed to handle regular expressions containing a singlequote; this has been fixed.

• BUG FIX (Secure Messaging only): If a user account is disabled, CanIt would sometimes issuea misleading error message. This has been fixed.


• POLICY CHANGE: Support for Debian Lenny (Debian 5.0) has been dropped. Debian Lennyhas not had security support for 5 years and no-one should be using it any more.

• NEW FEATURE: Custom Rules, Delivery Policy Rules, and Secure Messaging rules can matchagainst URLs in the message and also against just the URL hostnames.

• NEW FEATURE: The log-indexer lets you search for logs with attachment filename extensionsby looking for “ext:XXX” in the List of Tests Hit field.

• NEW FEATURE: Most rules have their IDs hyperlinked to a pre-filled log search form that willfind those rule hits in the logs.

• NEW FEATURE: The URL Proxy landing page lets a user vote that a URL is malicious.

• IMPROVEMENT: CanIt will de-proxy URLs in outbound message. Thus, if an incoming mes-sage had its URLs rewritten and it is forwarded out, the rewriting will be undone.

• IMPROVEMENT: The Verification Server page warns if a domain has an LDAP user-lookup(rendering the verification server superfluous.) A simlar warning appears on the Domain Map-ping page.

• IMPROVEMENT: <form> tags are disabled when displaying HTML messages.

• MINOR IMPROVEMENT: CanIt rewrites the command line of mimedefang-multiplexor pro-cesses to give more detailed information about what a given scanner is doing. These details arevisible to the “ps” UNIX command.

• MINOR IMPROVEMENT: Hovering over a non-zero rule hit count will display the date of thelast rule hit.


http://www.roaringpenguin.com


• MINOR IMPROVEMENT: If CanIt detects PDF file “foo.pdf” that contains one or more URLs,it adds a pseudo-filename foo.pdf.contains url to the list of attachment names. You can test forthis in a custom rule.

• API CHANGE: The Archived Mail search API returns messages in ascending timestamp orderrather than descending.

• API CHANGE: The GET /incident/ API call returns a list of URLs found in the incident bodyin a hash member named ’urls’.

• IMPROVEMENT: The Filename Extension rule matcher has been improved and many bugswere fixed.

• BUG FIX: In rare cases, a log search could return out-of-realm results. This has been fixed.

• BUG FIX: Correct many mis-spelled words in the manuals.

• BUG FIX: In rare situations, the Compound Rule compiler engine could generate syntactically-invalid Perl code; this has been fixed.

• BUG FIX: Some log searches would generate invalid SQL code; this has been fixed.

• BUG FIX: Invalid recipients could in some cases be counted in the provisioning data. This hasbeen fixed.

• BUG FIX: Several bugs in the Compound Rules GUI were fixed, most notably a bug that couldsometimes add extraneous clauses to the rule after it is saved.

• BUG FIX: CanIt Storage manager could fail when transferring huge amounts of data (more than4GB at once.) This has been fixed.

• BUG FIX: CanIt would sometimes display nothing when trying to view messages with invalidutf-8 encodings. It now displays the utf-8 replacement character in place of invalid code se-quences.

• BUG FIX: Trying to create a compound rule with “Arrival Day of Week” would fail with anerror. This has been fixed.

• BUG FIX: Accidental double-decoding caused the “top 15” Bayes words to be rendered asgibberish (“mojibake”) in the Incident Details report if they contained non-ASCII characters.This has been fixed.


• NEW FEATURE: Custom Rules (and similar rules like Archive Rules, etc.) as well as Archiveand Log Searches can search within a network range (expressed as a CIDR). For example, youcan say: IF Relay Address Is In Network 192.168.5.0/24 THEN ...


239

• NEW FEATURE: Filename Extension rules work for “multi-extension” files. For example,given an attachment named “foo.doc.html.exe”, CanIt now searches the following rules in order:doc.html.exe, html.exe and exe and takes the first one hit.

• IMPROVEMENT: CanIt detects files embedded in PDF attachments and adds their informationto the list of attached filenames.

• BUG FIX (CanIt-Domain-PRO only): The ConnectWise integration module would update theEffectiveDate each time it ran. This is not correct and the behavior has been removed.

• BUG FIX (Appliance Only): The nightly cron job ensures that a “postmaster” alias is alwaysdefined.

• BUG FIX (Secure Messaging): If an outbound email is quarantined and then released, SecureMessaging rules would not be applied. We STRONGLY recommend that everyone using SecureMessaging upgrade to 10.1.4.

• BUG FIX: In an installation with a sharded database, the “Top Rules” report would report “NoData”. This has been fixed.

• BUG FIX: Several small bugs in the Compound Rule GUI were fixed.

• BUG FIX (Archiver): We put a time limit on how long indexing an archive message can take toavoid a huge message stalling everything.


• MINOR IMPROVEMENT: You can include recipients in Pending Notification details using the%{recipients} tag.

• MINOR IMPROVEMENT: The Custom Rule editor has been tweaked with a few usabilityimprovements.

• BUG FIX: Due to an error, realm administrators could not update domain routing for theirsubrealms using the API. This has been fixed.

• BUG FIX: Due to a programming error, realm administrators could not specify the Realm fieldin a log search unless they were also the site administrator. This has been fixed.

• BUG FIX: The sort links on the Log Search page did not work. This has been fixed.

• BUG FIX: The CanIt Storage Manager server would fall out of sync with the client during trans-fers of large files (over 2GB). This has been fixed. *** THIS POTENTIALLY CAN CAUSELOSS OF STATISTICS SO WE URGE AN UPGRADE ***

• BUG FIX: Many HTML validation errors and unbalanced tags were fixed.

• BUG FIX: In Policy Rule Actions, the %{connecting relay address} and%{connecting relay hostname} were accidentally swapped (so you’d get the name whenyou asked for the address and vice-versa). This has been fixed.



• BUG FIX: The sample pgbouncer.ini file was adjusted so pgbouncer listens only on the loopbackaddress. This closes a potential security hole if you use the sample file as-is.

• BUG FIX: The Custom Rule Compiler attempts to diagnose and optimize regular expressionswith useless unbounded quantifiers (such as rewriting “.*foo.*” to simply “foo”, for example.)It also rejects regular expressions such as “foo.*bar.*quux” in body matches since regular ex-pressions with multiple unbounded quantifiers can take exponential run-time.

• BUG FIX: The code to create a zip file of archived mail would lose track of the actual queryand fail. This has been fixed.

• DOCUMENTATION FIX: The fact that Custom Rule matches are case-insensitive was docu-mented.

• DOCUMENTATION FIX: The security implications of someone gaining access to the Post-greSQL database were more fully explained in the Administration Guide “Firewall” section.


• MAJOR IMPROVEMENT: The Compound Rule editor has been improved significantly, per-mitting modification of any part of a compound rule as well as addition or deletion of clausesanywhere in the rule. This improvement has also been carried over to the Log Search andArchive Search interfaces.

• IMPROVEMENT (CanIt-Domain-PRO only): The Archive and Log Search interfaces allowyou to specify a realm or stream as “rname:sname” and internally map that to “(realm IS rnameAND stream IS sname)”

• MINOR IMPROVEMENT: The LDAP user-lookup test code provides additional hints if it en-counters various Active Directory error codes.

• BUG FIX (Log Search only): A substring match on the Message-ID field did not work; this hasnow been fixed.

• BUG FIX: When downloading a raw MIME message, lines are now terminated with \r\n(Carriage-Return / Newline) to mimic how they are transmitted on the wire. While the old\n-only format did not cause trouble for UNIX systems, some Windows software doesn’t workunless lines are terminated with \r\n.

• BUG FIX: Changes to user-lookup settings are now recorded in the Audit Trail table.

• BUG FIX: A regression introduced in 10.1.1 meant that CanIt would on rare occasions fail toremove existing voting links from messages; this has been fixed.

• BUG FIX: A message coming in over IPv6 that is subsequently streamed did not reverse-resolvethe oringal IPv6 sending relay. This has been fixed.

• BUG FIXES: A number of PHP warnings have been fixed.


241


• NEW FEATURE: A new test called Shortener404 fires on shortened URLs that return a 404Page Not Found. As a bonus, we collect URLs returned by URL shorteners and they are madeavailable for checking against the Known Phishing URL list.

• IMPROVEMENT (CanIt-Domain-PRO only): The “rlm=xxx” parameter is included in allURLs in CanIt-generated notifications; this keeps theming and branding active even on thelogin page.

• IMPROVEMENT: The Custom Rules page has been enhanced with a “quick rule” entry formthat lets you quickly create simple one-clause Custom Rules.

• IMPROVEMENT: The Verification Server code recognizes several more Temporary Failurecodes that really should be converted to Permanent Failures.

• BUG FIX: If you had quick-linked the old Custom Rules page, that link broke with no way todelete it. Upgrading CanIt fixes this edge case.

• BUG FIX: The “filter” in the Custom Rules page matches both comments and the actual ruletext itself, rather than only comments.

• BUG FIX: The old-style Custom Rules had field/relationship combinations not supported bythe new-style rules, causing certain rules not to be displayed. This has been corrected.


• POLICY CHANGE: The old-style Custom Rules are gone. Compound Rules have been re-named to Custom Rules and they replace Custom Rules. On upgrade, all existing old-styleCustom Rules are converted to the new-style rules. Note, however, that rule hit-counts for theold-style custom rules are lost on conversion.

NOTE INCOMPATIBILITY

• POLICY CHANGE: The “Header Sender” and “Domain of Header Sender” fields in customrules are gone. They are almost never useful; instead, you should use “Header From” and“Domain of Header From”. On upgrade, existing rules referring to the Sender: header arerewritten instead to refer to the From: header


• NEW FEATURE: When viewing a message from the quarantine, archive or Secure Messaginginterface, you can now download all attachments at once in a ZIP file.

• NEW FEATURE: Compound Rules, Archiver Rules and Secure Messaging Rules have fieldsand macros for the email address of the first recipient and the domain of that address.

• NEW FEATURE: If CanIt detects an encrypted ZIP or MS Office document file, it adds an“.encrypted” filename to the list of filenames seen. For example, if an email contains an



attachment called “foo.zip” that is an encrypted ZIP file, then the filenames “foo.zip” and“foo.zip.encrypted” will be reported.

• NEW FEATURE: Message subjects are now logged in the short-term (hourly) statistics data;you can run reports to see the top message subjects.

• IMPROVEMENT (Archiver): The Archive Importer can import .eml files as well as mbox andPST files. It also records its progress in a journal file so an interrupted archive import can beresumed from where it left off.

• IMPROVEMENT: In tag-only mode, a virus scanner hit adds a test name calledVIRUS(name of virus) to the list of tests hit.

• IMPROVEMENT (Appliance only): You can search logs with one-minute granularity insteadof just daily granularity.

• IMPROVEMENT: The AutoOpen macro detector now examines Microsoft Office files that arewithin ZIP files.

• MINOR IMPROVEMENT: canit-api-wrapper ignores the case of theGET/PUT/POST/DELETE keyword.

• BUG FIX: Poor performance in rule hit-counting has been improved dramatically.

• BUG FIX: Rules : Plugins refuses to display anything for the ’*’ pseudo-stream rather thandisplaying meaningless settings.

• BUG FIX: /realm/stream/setting API call did not respect the “Notification” permission; this hasbeen fixed.

• BUG FIX: Setup : Cluster Management : View Active Database Queries failed for versions ofPostgreSQL newer than 9.1; this has been fixed.

• BUG FIX: Perl API client correctly logs out if the client object goes out of scope.

• BUG FIX: The failover code would sometimes complain that PostgreSQL was running when infact it was not; this has been fixed.

• BUG FIX: The failover code did not interpret the “use pgbouncer” configuration item as aboolean. This has been fixed.

• BUG FIX: A few places in CanIt that could hang forever waiting on network connections havehad timeouts imposed.

• BUG FIX: ConnectWise integration: Implemented a workaround for a bug in ConnectWise’sAPI.

• BUG FIX: Correctly handle LDAP servers that return an error code rather than an empty resultlist for a query that returns no results.


243


• NEW FEATURE (Archiver): The CanIt site administrator can perform an Archive Search andthen redeliver some or all of the search results in a batch operation.

• IMPROVEMENT: You can add multiple DKIM keys and selectors for a given domain. Thismakes key rollover easier.

• IMPROVEMENT: Immediate Locked Addresses can use either “+” or “ . ” as the delimiter.Since some web forms won’t accept a + in an email address, use . instead in that situation.

• CHANGE: The terms “whitelist” and “blacklist” have been replaced with various forms of“block” and “allow”.

• BUG FIX: Much better error logging during cluster communication operations, especially whenBayes files are synchronized across the cluster.

• BUG FIX: Use more appropriate SMTP reply codes and DSN’s in certain situations.

• BUG FIX: Better recognition of MS Office attachments for macro-detection.

• BUG FIX: Suppress some irrelevant SSH messages during intra-cluster communication.

• BUG FIX: Lower-case all email addresses in the Users table so that they work correctly as ValidRecipient addresses.

• BUG FIX: Include original message headers when replying to an archived message or a SecureMessage.


• BUG FIX: A few places in the Web interface would forget the “rlm” and “s” parameters whennavigating from page to page; this has been fixed.

• BUG FIX: In certain very unusual cases, aliasing could fail for locally-generated mail. This hasbeen fixed.

• BUG FIX: Immediate Locked Address functionality has been fully enabled in the GUI and theyhave been made case-insensitive.


• NEW FEATURE: The Locked Address feature has been enhanced with a variation called “Im-mediate Locked Addresses” that lets you create a locked address without informing CanIt be-forehand.

• NEW FEATURE: An SMTP Server Test module lets you run a test against a back-end SMTPserver and receive helpful debugging output as well as some explanatory comments.



• POLICY CHANGE (CanIt-Domain-PRO only): The Setup > Domain Routing page now showsdomains associated with the current realm and all of its subrealms. Before, it would showeverything (for the site administrator) and only those domains in the current realm (for realmadministrators).

• MINOR NEW FEATURE: You can limit the ConnectWise and Autotask updates to run onspecific days of the month.

• CHANGE: If the country of a sending server can be determined, that country’s flag is displayedin the “Sender” column of the quarantine display.

• IMPROVEMENT: Delivery Policy Rules have been extended with new actions and made moreflexible with the ability to use macros inside action parameters.

• IMPROVEMENT: The Valid Recipients Table now also automatically includes aliases, explicitstream mappings, and in-realm email addresses in the users table. This greatly reduces theamount of duplicate data you need to enter into CanIt to make use of the Valid Recipientsfeature.

• IMPROVEMENT: Pending Notifications include a hazard icon near the subject of quarantinedmessages if they contain a held filename extension.

• COSMETIC FIX (CanIt-Domain-PRO only): If a domain is associated with many Known Net-works, only the first four are shown to avoid huge amounts of data in Administration > Provi-sioning.

• BUG FIX: Removal of pre-existing inline HTML voting links is now far more reliable thanbefore, even in the face of mangling by mail readers.

• BUG FIX: The “is not” relation in Archive Search rules would generate invalid SQL, causingan exception to be thrown. This has been fixed.

• BUG FIX: The SRS feature would sometimes cause locally-generated delivery status notifica-tions to be lost completely. This has been fixed.

• BUG FIX (CanIt-Domain-PRO only): The RSS URL Base URL was taken from the “base”realm instead of the correct realm.

• BUG FIX: The failover setup code would create the recovery.conf file with wrong ownership.This has been fixed.


• MAJOR IMPROVEMENT (CanIt-Domain-PRO Only): Autotask integration has been re-vamped completely; the new code supports any billing cycle (not just monthly billing on thefirst of the month) and all possible CanIt products including inbound and outbound filtering,Secure Messaging and Archiving.


245

• IMPROVEMENT: Additional Delivery Policy actions that allow changing the domain of thesender.

• IMPROVEMENT: Additional tests for Compound Rules including arrival-time-based tests.

• IMPROVEMENT (Secure Messaging Only): All outbound secure messages are stored in thesender’s “Sent” folder—not just messages that are created within the Secure Messaging inter-face.

• UPDATE: Update ClamAV from 0.99.1 to 0.99.2.

• COSMETIC FIX: If an incident is held because of a Filename Extension rule, it is annotatedwith a little hazard icon.

• COSMETIC FIXES: Minor tweaks to theme CSS files.

• BUG FIX: The DMARC code was incorrectly using the DMARC record associated with theDKIM “d=xxx” tag rather than the RFC5322.From domain. This has been fixed.

• BUG FIX: The code to retrieve held attachments could fail with Internet Explorer if the attach-ment filename had accented characters. This has been fixed.

• BUG FIX (Secure Messaging Only): Secure Messaging users can log in with their CanIt cre-dentials. This was partly implemented in the previous release, but did not work properly in allcases.

• BUG FIX (Appliance Only): The log-line parser could sometimes misinterpret message IDscontaining a “%” followed by two hex digits.

• BUG FIX (Appliance Only): A fatal PHP error in log searching has been fixed.

• BUG FIX: In previous release, the domains associated with a Known Network were displayedin random order. They are now properly sorted.


• MAJOR NEW FEATURE: A new Delivery Policy Module lets you create rules that affect howmail is delivered after CanIt has scanned it and is about to deliver it.

• MAJOR NEW FEATURE (CanIt-Domain-PRO Only): CanIt can integrate with ConnectWiseto automate billing.

• POLICY CHANGE: In addition to looking up exact matches in the Address-to-Stream tablefirst, CanIt now also looks up local parts with wildcard domains. In other words, the order oflookups for “[email protected]” is now:

1: [email protected] 2: user@* 3: Whatever Domain Mapping is defined for example.com

• IMPROVEMENT (Debian Appliances Only): Sender, Network, Domain, Filename Extensionand MIME rules all have statistics on how many times they hit.



• IMPROVEMENT (Secure Messaging Only): If a recipient already has a CanIt account, he orshe can log into that account to access the Secure Messaging portal rather than having to createa completely separate account.

• IMPROVEMENT: The URL Proxy feature has a third option for whether or not to wrap a URL.In addition to “Wrap” and “Don’t Wrap”, you can now specify “Wrap if Tagged as Spam” whichwraps URLs only if a message was tagged as spam in tag-only mode.

• IMPROVEMENT: In Pending Notifications (HTML Format), the full name in the From: headeris included, in addition to the email address.

• IMPROVEMENT: A new “Iceberg” Web interface theme has been added to the standard themesshipped with CanIt.

• COSMETIC FIX: Minor CSS fixes to the default RP-Web theme.

• BUG FIX: canit-failover-init.pl could fail on versions of PostgreSQL higher than 9.1 that alsouse tablespaces. This is a very unusual configuration and not likely to be a problem in practice.

• BUG FIX: Releasing an incident by clicking on the notification email link could appear to work,but actually fail if you are logged in as a user who lacks access to the original stream. This hasbeen fixed.

• BUG FIX: Many more filename extensions are recognized as MS Office documents and scannedfor macros.

• BUG FIX: The one-time key encryption method could fail if you tried replying to an encryptedmessage. This has been fixed.

• BUG FIX: Log query searches header From: field when sender is specified.


• NEW FEATURE: Users with sufficient privilege can request specific domains to be exemptedfrom URL-proxying.

• NEW FEATURE: CanIt can detect PDF files that contain JavaScript; if found, it adds thepseudo-filename “canit js found.js in pdf” to the list of attachments. (CanIt Appliances andHosted CanIt only.)

• NEW FEATURE: CanIt can extract URLs from PDF documents for testing against the known-phishing list. (CanIt Appliances and Hosted CanIt only.)

• POLICY CHANGE: Roaring Penguin no longer supports new source or RPM installations; allnew installations must use our ISO or be converted to an appliance from Debian. The sourceand RPM packages will continue to be maintained for the purpose of upgrading legacy CanItinstallations.


247

• POLICY CHANGE: In tag-only mode, CanIt would unconditionally add a tag with the “detail”field. It no longer does that; instead, you should explicitly include “%e” in your tag string if youwant the detail included.

• IMPROVEMENT (Archiver): When replying to a message within the Archived Mail interface,you can request CanIt to Cc: the sender.

• MINOR IMPROVEMENT: Filename Extensions of attachments are logged in the “what=xxx”mail log line as the “attach types=xxx;yyy;zzz” keyword.

• SECURITY IMPROVEMENT: Make sure all local passwords use the stronger MD5-style en-cryption rather than traditional UNIX-style password encryption.

• MINOR IMPROVEMENT: Implement “Show Changes” on Setup > Known Networks page.

• BUG FIX: If a message has been forced to a stream because of a Known Networks entry, weskip SPF, DKIM and DMARC lookups.

• BUG FIX: In a few places in unusual circumstances, an uncaught exception could kill the scan-ning process and tempfail mail. This has been fixed.

• BUG FIX: The “Contains Credit-Card” Compound Rule component was a bit lax and couldfalsely claim an email contains a credit card number; the code has been tightened up to reducefalse-positives.

• BUG FIX: Several other minor filtering and GUI bugs were fixed.

• BUG FIX: If a file has trailing space (for example, “malware.js ”), then CanIt ignores the trail-ing space when applying Filename Extension rules. In this example, the extension would beconsidered to be “js”.


• MINOR NEW FEATURE: DMARC can now be run in a “Quarantine” mode. In this mode, itquarantines messages that hit DMARC “reject” or “quarantine”. The “Enforce” mode is stricterand it rejects messages with a DMARC result of “reject”.

• IMPROVEMENT: A new pulldown menu style is available for the RP-Web and Postmodernthemes; these menus let you navigate with only one click to any first- or second-level menupage. By default, pulldown menus are disabled but users can enable them under Preferences.

• IMPROVEMENT (CanIt-Domain-PRO only): Autotask integration has been improved; if aContract has a purchase order associated with it, the PO number is copied into the invoiceposted by CanIt.

• BUG FIX: DMARC results are always added to the incident report, even in dry-run mode.

• BUG FIX: DKIM and DMARC would sometimes incorrectly use the address in the Sender:header rather than the From: header. This has been fixed.



• BUG FIX: Minor cosmetic errors in the Web interface HTML were corrected.

• BUG FIX: The login page templates now respect theme customizations.


• MAJOR NEW FEATURE: CanIt now supports testing for DMARC policy. However, DMARCreporting is not yet implemented.

• MAJOR NEW FEATURE (Secure Messaging only): You can now elect to use a new one-timekey encryption scheme. In this scheme, the decryption key is encoded in the URL sent to theoriginal recipients. It is impossible to decrypt the message without possessing the URL, even ifsomeone obtains the recipient’s Secure Messaging credentials.

• POLICY CHANGE: By default, non-administrative users can no longer vote URLs as fraud-ulent. However, this permission can be granted to them under Administration : Permissionsshould you deem it appropriate.

• CHANGE: We no longer offer Red Hat Enterprise Linux 5 RPM packages. We still offer RPMsfor Red Hat Enterprise Linux 6.

• MINOR IMPROVEMENT: If the “From:” header address is different from the envelope sender,an additional “header [email protected]” key pair appears in the “what=...” linelogged by CanIt.

• MINOR IMPROVEMENT: The Known Networks page is now paginated, yielding much fasterdisplay times for sites with many Known Networks entries.

• MINOR IMPROVEMENT (CanIt-Domain-PRO only): The Provisioning page displays KnownNetworks associated with outbound relaying for a given domain.

• MINOR IMPROVEMENT: (CanIt-Domain-PRO only): A new Provisioning History pageshows provisioning statistics per realm over time.

• IMPROVEMENT: The LDAP code is much more intelligent about guessing a user’s primaryemail address from the LDAP attributes that are returned. In particular, it understands Mi-crosoft’s convention that the primary address is prefixed by SMTP: (upper-case) as opposed tosmtp: (lower-case).

• MINOR IMPROVEMENT: The API call GET /info returns some additional info for root-privileged users (in CanIt-Domain-PRO, only base-realm root-privileged users.)

• MINOR IMPROVEMENT: The /xauth API call accepts a “logout redirect” parameter. Thispermits you to redirect to a page that logs a user out of an entire single sign-on system when heor she logs out of CanIt.

• MINOR IMPROVEMENT: The Verification Server code attempts to detect a back-end serverthat is imposing “tarpitting”. It raises an anomaly if tarpitting is suspected.


249

• CHANGE: The maximum possible timeout for Verification Server checks has been increased to120 seconds from 30 seconds.

• BUG FIX: The Anomaly Notification nightly task would sometimes use templates from thewrong realm when composing its email. This has been fixed.

• BUG FIX: In several places, the code incorrectly assumed that the force to stream Known Net-works attribute applied on inbound-only hosts when in fact it does not. This bug has been fixed.


• NEW FEATURE (Archiver only): Users with the appropriate permission can compose brandnew email messages within the Archiver web interface. This lets you keep doing business ifyour back-end mail server is down.

• IMPROVEMENT (CanIt-Domain-PRO only): The provisioning report under Administration :Provisioning is now calculated much more quickly than before.

• IMPROVEMENT: If an email is rejected because of SPF “fail”, the SPF error message is in-cluded in CanIt’s 5xx reply.

• POLICY CHANGE: Normal end-users can be granted permission to make URL Proxy rules.Before, only administrators could do so.

• DEPRECATION: The old and deprecated canit-api-client command-line tool has been removed.Instead you should use the new canit-api-wrapper tool. As part of this change, the /introspectionAPI call has been removed.

• MINOR IMPROVEMENT: The report pages with the long list of possible classifications let youset, clear and toggle all the classifications with one mouse click.

• BUG FIX: SPF/DKIM VBR lookups would be ignored in favor of a wildcard SPF/DKIM rule.This has been fixed so that wildcard rules do not override VBR rules.

• BUG FIX: A long-standing bug in the Permissions user-interface that appeared to make permis-sions change by themselves has been fixed.

• BUG FIX: The URL proxy landing page now checks for both the original URL and the baseURL (no query parameters) in the list of known-phishing URLs.

• BUG FIX: CanIt would sometimes raise an anomaly complaining that a domain’s DKIM DNSrecord does not match the DKIM key, when in fact the two do match. This has been fixed.

• BUG FIX: The URL proxy correctly handles the <base> tag.

• BUG FIX: The CanIt Storage Manager would sometimes log incorrect storage-manager trafficstatistics; the statistics-logging module has been overhauled to correct this.




• NEW FEATURE: A Compound Rule can refer back to the list of other rules hit so far, allowingthe powerful composition of “meta rules”. See the manual for details.

• NEW FEATURE (Appliance Only): The log lines resulting from a log search can be down-loaded as plain-text.

• IMPROVEMENT: The OfficeMacroAutostart test was split into three tests that look forAuto Open, Document Open and Workbook Open macros in MS Office documents; you maywish to score the different macros differently.

• POLICY CHANGE: Automatic updates are disabled for appliances running PostgreSQL earlierthan 9.0. Such appliances can still be upgraded by hand.

• BUG FIX: The alias-replacement mechanism would sometimes deliver to both the original andthe aliased address if the original address was mixed-case. This has been fixed.

• BUG FIX (Appliance only): An error parsing a logline containing “forced into stream xyz:abc(derived from xyz:@@)” has been fixed.


• NEW FEATURE: A new test plugin called OfficeMacroAutostart detects macros in MS Officedocuments that are designed to start as soon as the document is opened. Such macros are highlysuggestive of macro viruses.

• PERFORMANCE IMPROVEMENT: On busy systems, the Archive Indexer background pro-cess can run with multiple concurrent indexing processes. See the Administration Guide docu-mentation of the configuration setting [ticker] index archived mail parallel indexers.

• BUG FIX: It is not possible to filter the top rule hits by domain, so remove the domain box fromthe corresponding report Web page.


• NEW FEATURE: Each domain associated with a Known Network may have its own separate“Force-To-Stream” entry. This provides additional flexibility for dealing with outbound mail.

• NEW FEATURE (CanIt-Domain-PRO Only): A new page Administration : Provisioning showsusage information in a way that is convenient for billing.

• MINOR IMPROVEMENT (Appliance Only): Hovering over the “Hits” column on rule pagesshows the date the rule last fired, assuming Hits is non-zero.

• BUG FIX: There were extensive fixes to the handling of Unicode data throughout CanIt. Thesefixes include safer handling of malformed messages with octets > 127 directly in the headers.


251

• BUG FIX (Appliance Only): Custom Rules now sort on the “Hits” column correctly.

• BUG FIX: Various parse errors in the pure-PHP MIME parser were fixed.

• BUG FIX: Fix deprecated way of using “crypt” function in PHP.


• MINOR IMPROVEMENT: If DKIM signing is set up, but no DKIM DNS records have beenpublished for a domain, the nightly cron job will raise an anomaly warning.

• BUG FIXES: Fixed a number of problems with UTF-8 data encoding on Debian Jessie, causedby an upgraded version of the PostgreSQL perl client library.


• MINOR NEW FEATURE: You can download the complete raw MIME message when viewinga held message.

• MINOR NEW FEATURE: The interval between Storage Manager latency checks is config-urable rather than being hard-coded at one hour.

• MINOR NEW FEATURE: “match”-type DNSBLs now permit the specification of ’X’ as anoctet; this acts as a wildcard that matches anything from 0 to 255.

• UPDATE: Our appliance ISO is now based on Debian 8 “Jessie” and we have Jessie packagesavailable to upgrade appliances to Jessie.

• END-OF-LIFE: 9.2.6 will be the last version for which Debian 5 “Lenny” packages will bemade available.

• BUG FIX (Appliance only): The log-searching page could lose track of the current search ifyou clicked on an arrow to sort results. This has been fixed.

• BUG FIX: CanIt now treats “permerror” as “error” for the purpose of SPF scoring.

• BUG FIX (Secure Messaging only): Properly quote the header From: full name.

• BUG FIX (Secure Messaging only): The subject could be truncated in the message display ifthe subject header was wrapped. This has been fixed.

• BUG FIX: Streamed messages could end up using the wrong IP address for DNSBL lookups.This has been fixed.

• BUG FIX: A typo prevented the xauth API call from working correctly without a ’redirect’parameter; this has been fixed.




• NEW FEATURE: CanIt now supports aliases of the form:

*@domain1.example.org ==> %[email protected]

which rewrites the domain part while keeping the original local part. This achieves so-calledDomain Aliasing.

• NEW FEATURE (Secure Messaging only): Administrators can disable Secure Messaging ac-counts. In addition, realm administrators can delete Secure Messages associated with theirrealms.

• IMPROVEMENT: The DKIM key-pair page lets you specify a DKIM selector. This allows forgraceful rollover of keys.

• IMPROVEMENT: The DKIM signature algorithm has been changed from rsa-sha1 to rsa-sha256

• UPDATE: SpamAssassin has been updated from version 3.3.2 to 3.4.1

• UPDATE: ClamAV has been updated from version 0.98.6 to 0.98.7.

• BUG FIX (CanIt Appliance only): The Log Indexer would index the source and destinationIP addresses as 127.0.0.1 for streamed messages, instead of using the correct external relayaddresses. This has been fixed.

• BUG FIX: CanIt::Sendmail’s SMTP timeout was too short, which could result in duplicatePending Notification messages on busy systems. This has been fixed.

• BUG FIX: A rounding problem could result in duplicate pending notifications for a given inci-dent even if the “Only notify me about new incidents” flag is set. This has been fixed.

• BUG FIX: Accepting an incident multiple times in quick succession could result in multiplecopies being remailed. This has been fixed.

• BUG FIX: A streamed message coming in originally over IPv6 could have the wrong hostnameassociated with the relay IP. This has been fixed.

• BUG FIX: The “Vote as Phish/Fraud” link did not actually permit voting on malicious URLs.This has been fixed.

• BUG FIX: The code to handle “Associated Domains” with Known Networks treated domainnames case-sensitively. This has been fixed.


• MAJOR NEW FEATURE: Users can mark messages as fraudulent and then specify whichURLs in the message look malicious. Administrators can add those URLs to the Known Ma-licious URL list; additionally, they are reported back to Roaring Penguin via our reputationsystem.


253

• MAJOR NEW FEATURE: CanIt can DKIM-sign outbound mail.

• NEW FEATURE (Appliance only): We package and enable code to pull down additional se-lected ClamAV signature sets.

• IMPROVEMENT: You can specify exactly which domains should be relayed from a givenknown network. (By default, if a known network has the ”allow relaying” flag on, then anymessage from that network is relayed regardless of the sender domain.)

• IMPROVEMENT: The URL Proxy feature can proxy the target URL of forms as well as ordi-nary links.

• IMPROVEMENT (CanIt-Domain-PRO appliance only): The Autotask integration code allowsyou to specify a minimum number of units to bill each month.

• IMPROVEMENT: The OfficeMacros test is better at detecting macros inside modern MS Officefiles.

• IMPROVEMENT: SMTP Extended Status codes have been changed to better reflect the nuancesof the response. For example, the nonexistent recipient code changed from 5.7.1 to 5.1.1 whichis more appropriate according to the RFC.

• IMPROVEMENT (Appliance Only): Log searching was made more flexible (additional opera-tors are now possible for various fields) and a few minor bugs were fixed.

• UPDATE: Update ClamAV from version 0.98.5 to 0.98.6.

• CHANGE (CanIt-Domain-PRO only): The API call to rename realms allows you to skip re-naming the realm in the statistics and log-index tables, which can be extremely time-consuming.However, if you choose not to rename in the statistics tables, then the realm’s statistics are lostas is the ability to search logs for the realm prior to the renaming.

• BUG FIX (Archiver only): Some usage reports would crash due to invalid SQL. This has beenfixed.

• BUG FIX: Messages auto-released from delayed streams simply sailed through the system.Now they are scanned as usual.

• BUG FIX: A race condition in the background task code could result in failures when usersattempted to set up aliases. This has been fixed.

• BUG FIX: An error in the Custom Rule evaluator could make some rules that use regular ex-pressions fail to match correctly. This has been fixed.

• BUG FIX: Anomaly details could contain the incorrect host name. This has been fixed.

• BUG FIX: Messages containing <tr> tags outside of a <table> could mess up the messagepreview page. This has been fixed.

• BUG FIX: The web theming system had a small error which made it impossible to change thecolor of menu text. This has been fixed.



• BUG FIX: Various errors in extracting URLs for the URL Proxy have been fixed.

• BUG FIX: If a stream is in tag-only mode, we change the wording of “Always Hold” to “AlwaysTag”; similar changes occur in various places in the GUI.

• BUG FIX: We add configuration items to /etc/mail/canit/sa-canit.cf to suppress SpamAssassin’sautomatic generation of “trusted” networks.


• MAJOR IMPROVEMENT: CanIt has a test to detect URLs on a Known Phishing URL list andadd points or block messages containing a malicious URL.

• MAJOR IMPROVEMENT: Anomalies now record each individual occurrence of the anomalyand (if the Log Searching component is installed) include links to relevant log lines.

• NEW FEATURE: CanIt has code to detect Microsoft Word documents that contain macros.You can then add points if one is found. This can help combat Word macro viruses, which areincreasingly used to compromise workstations.

• API IMPROVEMENT: The API call “GET /realm/@@/stream/@@/incidents” call permits fil-ter conditions to be supplied to limit the list of incidents that are returned.

• IMPROVEMENT: CanIt parses the text of HTML attachments even if they have type“application/octet-stream”, as long as their filename ends with .htm or .html and they appearto contain HTML content.

• IMPROVEMENT: The URL Proxy feature can now proxy HTML form targets as well as normal“<A>” links.

• IMPROVEMENT: A special notation “>ext” allows you to create Filename Extension rules thatapply only to files found within archive files. Thus, for example, an extension rule of “>zip”would apply only to a zip file contained in another zip file or some other type of archive.

• IMPROVEMENT: IPv6 geolocation has been improved to include latitude, longitude, city andregion if such data is available.


• POLICY CHANGE: For messages over 9MB in size, CanIt checks against a verification servereven if the recipient is cached as valid. This avoids backscatter with servers that reject largemessages.

• COSMETIC IMPROVEMENT: If a whitelist is ignored due to SPF fail/softfail, the Web inter-face links to a page that explains what happened.

• BUG FIX: Data supplied for LDAP queries is escaped to avoid inappropriately passing wildcardcharacters back to the LDAP server.


255

• BUG FIX: The Compound Rule entry code now correctly validates entries that are supposed tobe IP addresses.

• BUG FIX: The API call “POST /vote” did not work correctly; this has been fixed.


• MAJOR NEW FEATURE: CanIt now supports SRS (Sender Rewriting Scheme), making itfeasible to use CanIt in front of a back-end server that performs SPF checks. Some configurationoutside of CanIt is required; see the manual for details.

• MAJOR NEW FEATURE: If you are using PostgreSQL’s streaming replication and have a hot-standby database, you can configure certain nodes to direct read-only queries to the hot-standbyserver instead of the primary database server. If you have a geographically-dispersed cluster,this can significantly improve performance by having most queries go to the database serverwith the lowest round-trip latency.

• MINOR IMPROVEMENT (CanIt-Domain-PRO only): The UDP transport for log forwardingpermits you to specify multiple destination hosts, each with its own port.

• MINOR IMPROVEMENT (Appliances only): The “Header From:” email address is logged,permitting log searches based on that field.

• COSMETIC CHANGE: French and German translations have been updated.

• COSMETIC CHANGE: The “Change Password” page has been reorganized to ask for your oldpassword first, which matches the way most such pages work.

• BUG FIX: An error in how Compound Rules (and Archiver and Secure Messaging rules) werecompiled down into Perl could result in warnings about undefined values in the mail logs. Thishas been fixed.

• BUG FIX: The URL Proxy code could sometimes double-encode MIME messages, breakingtheir display. This has been fixed.

• BUG FIX: If a message is relayed from a friendly host, we do not use the HELO information asBayes tokens since it is not a reliable indicator of ham/spam.

• BUG FIX: The rate-limiting code could leak rate-limiting information across streams andrealms. This has been fixed.

• BUG FIX: Logins with a full email address but mixed-case domain failed because the domainlookup was case-sensitive. This has now been made case-insensitive.

• BUG FIX (Archiver and CanIt-Domain-PRO only): The archive zip file page now shows zipfiles in the current realm, not necessarily the realm of the logged-in user. This only makes adifference for realm administrators who have subrealms and switch into them.

• BUG FIX: The PHP code understands a wider variety of MIME “charset=xxx” parameters,fixing display problems for certain messages.




• BUG FIX (Appliance Only): Fix a typo that broke upgrades against versions of PostgreSQL <8.4.

• BUG FIX: Fix edge-case in which transitioning from the old hash-based incident detectionalgorithm to the new incident-ID based algorithm could sometimes cause existing incidents notto display in the Web interface.


• POLICY CHANGE: If a CanIt cluster member is *not* marked “Outbound” in the ClusterMembers Table, then the force-to-stream Known Networks attribute is ignored.


The above policy change may change how force-to-stream works on your cluster.

• POLICY CHANGE: When CanIt logs a message subject in the “subject=XXX” field, it alwaysencodes the subject as UTF-8, regardless of the original encoding.

• MAJOR CHANGE: The global setting G-600 ”Send tempfail indications for suspect messages”has been removed and implicitly defaults to “Never”. Previous versions of CanIt used a hash-ing scheme to detect message retransmissions; this could fail in rare edge-cases. The hashingscheme has been removed and incident creation is now far more reliable As a side-effect, youcan now re-open and accept an erroneously-rejected incident and CanIt will deliver the message.

• MAJOR NEW FEATURE: Compound Rules have been enhanced with additional fields andrelations, as well as a macro feature that lets you specify things like “envelope sender” in thedata box; this permits extra flexibility when creating Compound Rules.

• MAJOR NEW FEATURE (Appliance Only): If you have the log indexing component installed,CanIt tracks how often custom rules and compound rules are hit. These statistics let you evaluatethe effectiveness of your rules and remove those that never or rarely hit.

• MINOR NEW FEATURE (Appliance Only): The log-searcher can search the “tests=X;Y;Z”log string so you can search for specific rule hits.

• MINOR NEW FEATURE: On a per-stream basis, you can request that rate-limiting rules applyto a stream even if mail was not forced into it by a Known Networks entry.

• MINOR NEW FEATURE (CanIt-Domain-PRO only): The force-to-stream field in Known Net-works lets you specify “@@:streamname” or “realmname:@@”. In either case, ’@@’ is re-placed with the realm of the envelope sender.

• IMPROVEMENT: The URL Proxy feature permits you to specify a separate template forsuspected-phishing URLs as opposed to normal URLs. You can also specify a different BaseURL for the URL Proxy page.


257

• IMPROVEMENT: The Pending Notification templates have additional substitution tags formore flexible formatting of dates and times.

• IMPROVEMENT: Display of MIME-formatted messages is supported on all platforms. Ap-pliances use fast C code to decode MIME messages; other platforms use a slower pure-PHPlibrary.

• IMPROVEMENT: The code to select Storage Manager nodes for writing attempts to ensure thatone copy of data is written in each geographical location if you provide location informationin the Cluster Members table. Selecting a node for reading continues to be ordered strictly bylatency.

• MINOR IMPROVEMENT: The API call for domain routing permits specifying the list of des-tination servers as a comma-separates string as well as an array.

• MINOR IMPROVEMENT (Secure Messaging): Secure Messaging can be configured via theAPI.

• MINOR IMPROVEMENT (Secure Messaging): You can specify a separate Base URL for theSecure Messaging portal than for the rest of CanIt.

• MINOR IMPROVEMENT: A config.php setting permits you to prohibit the “Queue All Ad-dresses” option for Verification Servers.

• MINOR IMPROVEMENT: Compound and Custom Rules with a score of zero are no longerevaluated at all. If you want to make test rules that don’t materially affect the score, assign ascore of 0.001 to force the rule to be run.

• COSMETIC CHANGE: The Known Networks “Don’t Tempfail Incidents” attribute has beenrenamed to the more descriptive “Friendly Host”

• BUG FIX: The cron job to expire old Bayes data now runs on all cluster members rather thanjust on the database server.

• BUG FIX/IMPROVEMENT: The Custom Rule code has been overhauled to make the codemuch simpler and Custom Rule evaluation faster.

• BUG FIX: If a rate-limit setting causes a rule to be created, the new rule is now entered into theaudit table and appears in “Show Changes”.

• BUG FIX: The code to test User Lookups now ignores internal records about back-end serversthat are down and always attempts to run the lookup.

• BUG FIX: If you choose to sort messages in pending notification by score, earlier versions ofCanIt could incorrectly ignore the ”Only show new incidents” flag. This has been fixed.

• BUG FIX: Sendmail accepts a RCPT command like this: RCPT To:<local@ example.com>The space confuses CanIt’s streaming mechanism, so CanIt is now hard-coded to reject RCPTcommands with whitespace in the domain part.



• BUG FIX: (Appliance Only): Searching logs by sender/recipient was case-sensitive; this hasnow been fixed to be case-insensitive.

• BUG FIX: The default DNS timeout for SPF checks has been set to a more reasonable 10seconds instead of 120 seconds.

• BUG FIXES: Numerous cosmetic bugs were fixed.


• IMPROVEMENT (CanIt-Domain-PRO only): User-lookups can be inherited by subrealms. Ifyou have a number of subrealms that all use the same LDAP settings, for example, this cangreatly simplify setup and reduce data duplication.

• CHANGE: We include a new command-line API client called “canit-api-wrapper”. We promisethat this one will last and not be deprecated, unlike the previous iterations of command-lineclients.

• POLICY CHANGE: Creating a Verification Server entry now defaults to “Queue Seen Ad-dresses” rather than “Tempfail” if the back-end server is down.

• POLICY CHANGE: If every machine has a “location” entry in the Cluster Members Table,CanIt attempts to use a Storage Manager node in each location first, and then any remainingwrites are done in order of measured latency.

• MINOR NEW FEATURE: You can explicitly log training links in the mail log so that you canvote messages based on data in the logs.


• BUG FIX (Appliances Only): The log-search page could lose track of the current query. Thishas been fixed.

• BUG FIX: CanIt would sometimes fail to import Custom Rules that had been exported by anolder version of CanIt. This has been fixed.

• BUG FIX: Storage Manager now implements timeouts when locking a file for append. It alsouses the TCP “keepalive” option to ensure that Storage Manager servers eventually exit if aclient machine crashes.

• BUG FIX: The URL Proxying feature could sometimes convert HTML entities to UTF-8 char-acters inappropriately. This has been fixed.

• BUG FIX: Perl warnings in the URL Normalizer have been fixed.


259


• POLICY CHANGE: SURBL rules are disabled by default since the SURBL maintainers mayrequire a commercial license to use their RBL. Instructions for re-enabling SURBL are includedin our dynamic ruleset; be sure you qualify for free usage or purchase a subscription beforeenabling SURBL rules.

• NEW FEATURE: You can supply a list of Bayes stop-words. These are words that will becompletely ignored by Bayesian analysis. This feature is designed to exclude common non-English words from Bayes in situations where much of your valid email is not in English.

• NEW FEATURE: Compound Rules, Archiver Rules and Secure Messaging Rules have built-intests to look for credit card numbers, Canadian social insurance numbers and US social securitynumbers.

• NEW FEATURE: We make use of a collaboratively-maintained list of known phishing URLs;our URL Proxy feature always proxies URLs on the list and also prevents users from clickingthrough to them.

• NEW FEATURE (Archiver and Secure Messaging only): When replying to a message via theWeb interface, you can now add attachments.

• POLICY CHANGE: URL proxying is enabled by default. However, on new installations weonly proxy URLs in the Known Phishing URL list.

• MAJOR IMPROVEMENT (Appliance only): Log-searching has been completely overhauled.You can now make complex searches with AND/OR/NOT combinations. You can namesearches and save them for later reuse.

• IMPROVEMENT: The code to import/export rules and settings from streams was completelyoverhauled. You can now import and export Quarantine Settings in addition to most types ofrules.

• MINOR NEW FEATURE: Per-realm theme customizations now also apply to the Secure Mes-saging interface.

• MINOR NEW FEATURE: Unofficial/contributed script canit-analyze-rule-hits.pl allows you toanalyze the hit rate of custom and compound rules.

• MINOR IMPROVEMENT: Audit trail (“Show Changes”) data persists for two years by defaultrather than the previous default of 90 days. You can also configure it to persist for up to 10 000days.

• MINOR IMPROVEMENT: Whenever we log a “PhishingAddress” rule hit, we now also logthe score of the hit.

• BUG FIX: If a domain completely lacks MX records, the “Bogus MX” test uses the A record(s),if any.

• BUG FIX: Several PHP warnings were eliminated.



• BUG FIX (CanIt-Domain-PRO only): The site administrator can now delete out-of-realm en-tries in the Valid Recipients Table.

• BUG FIX: The “Report Time Span” on Classification reports spans the time for which data isavailable rather than just the time from the first to the last event actually included in the report.This gives a more accurate picture.

• BUG FIX: An edge-case could occur in which a pending incident was created, but CanIt did notrealize it needed to send a Pending Notification. This has been fixed.

• BUG FIX (Appliance only): If a message is accepted or rejected directly by Sendmail withoutany CanIt log lines, the log parser assigns a resolution of “accepted” or “rejected” based onSendmail’s delivery status code.

• BUG FIX (Secure Messaging only): If a user clicks on the registration link but has alreadyregistered for a Secure Messaging account, CanIt now presents the normal login form.

• BUG FIX (Secure Messaging only): An extremely rare edge-case in which CanIt could createan invalid link for a secure message has been fixed.


• IMPORTANT SECURITY FIX: A cross-site scripting vulnerability in the URL Proxy page hasbeen fixed. *** ALL 9.1.2 USERS SHOULD UPGRADE ***

• SECURITY IMPROVEMENT: If you are running PHP 5.2.0 or newer, CanIt sets the“HttpOnly” flag on its cookie, which may help mitigate cross-site scripting attacks.

• NEW FEATURE (CanIt-Domain-PRO only): The “force to stream” feature lets you specify astream of the form: “realmname:@@”. The “@@” part is replaced with the realm of the enve-lope *sender* address. This can be used (for example) to implement per-domain disclaimers onoutbound mail.

• IMPROVEMENT: On Debian Squeeze and Wheezy appliances, and on all platforms where the“lsar” program is available, CanIt can look inside many different types of archives for filenameextensions. The list includes ZIP, RAR, and tar files. Previously, CanIt could only look insideZIP files.

• BUG FIX (CanIt Appliance only): When canit-setup-appliance enumerated available timezones, it would ignore ones that were symbolic links. This has been fixed.

• BUG FIX: LDAP lookups now have an overall timeout applied. Previously, if the connectionsucceeded but the LDAP server never responded, the scanning process would hang for a verylong time.

• BUG FIX (CanIt Archiver only): The archive importer program would fail unless you specifiedthe –force realm and –force stream flags. This has been fixed.


261

• BUG FIX (Secure Messaging only): When fetching a secure message, the fetch would be loggedseveral times. This has been fixed so each fetch is logged only once.

• BUG FIX: CanIt uses a UTF-8-aware word-wrapping function. This should avoid display prob-lems in the quarantine display for messages with very long subject lines.

• BUG FIX: The URL Proxy feature handles URLs with trailing punctuation in plain-text mes-sages in a way that preserves the original intent better.

• BUG FIX: If you add a custom header to all messages through CanIt, then CanIt replaces anyexisting custom header with the same name rather than adding a second custom header.

• BUG FIX: The URL Proxy administration page would fail if you switched to the “*” pseudo-stream; this has been fixed.

• BUG FIX: 9.1.2 would fail on PHP installations that lacked the mbstring extension. This hasbeen fixed so that CanIt continues to work, albeit in a somewhat degraded fashion.

• BUG FIX: The URL Proxy code could fail on Red Hat Enterprise Linux 5 because it used PHPfeatures lacking on that system. This has been fixed.

• BUG FIX: If the X-CanIt-Geo: header or a CanIt custom header has a character with the high-bitset, the header was not properly MIME-encoded. This has been fixed.

• MINOR BUG FIX: On source and Red Hat installations, permissions were loosened somewhatso the URL Proxy feature can use the geolocation database.

• MINOR BUG FIX: Several PHP warnings have been suppressed.

• COSMETIC FIX: Translation templates that are no longer used have been removed from theSetup : Templates page.

• COSMETIC FIX: The order of options to dispose of messages in the quarantine display hasbeen changed to group all “accept-type” options together followed by all “reject-type” options.


• MAJOR NEW FEATURE: CanIt can wrap URLs in email messages to take users to a landingpage warning them not to supply sensitive information. They can then click on a link to go to theoriginal URL. This may help reduce the success rate of phishing attacks. See “URL Proxying”in the Administration Guide.

• IMPROVEMENT: The Pending Notification email message can be templated to a larger degree,permitting larger changes to the appearance and content than was possible before.

• MINOR IMPROVEMENT: A preference controls whether CanIt displays a formatted or unfor-matted message by default when previewing a quarantined message.

• MINOR IMPROVEMENT: SPF and DKIM rules now have a “Comment” field.



• MINOR IMPROVEMENT: Choosing a “Vertically-Compact Trap Display” turns off some moreunnecessary line-breaking.

• BUG FIX: Permissions have been fixed so that a read-only user cannot create Periodic Reports.

• BUG FIX: The Bayes tokenizer now ignores tokens shorter than 3 characters except for ideo-graphic character sets like CJK Unified Ideographs.

• BUG FIX: Previously, CanIt would round SpamAssassin scores to one decimal place. This wasa bit too coarse; we now keep two decimal places of precision.

• BUG FIX (Appliance only): Fix a bug in the code that enumerated possible time zones.

• BUG FIX: If you specify a custom header and the header already exists on the incoming email,replace the header rather than adding a second one.

• BUG FIX: Use UTF-8-safe code to wrap the subject display in the Quarantine display.


• MAJOR IMPROVEMENT: If CanIt has a complete message held locally, then the messagepreview in the quarantine display formats the MIME message correctly instead of showing rawMIME. NOTE: This feature is available only on our Debian-based appliances and on HostedCanIt.

• IMPROVEMENT: The RSS feed includes a link to whitelist the sender of a trapped incident.

• MINOR IMPROVEMENT: The rendering of MIME messages within the Web interface hasbeen improved.

• IMPROVEMENT: The two flavors of relay address have been renamed to “Connecting RelayAddress” and “Sending Relay Address” and the definitions in the manual have been clarified:The Connecting Relay is the other end of the SMTP connection, while the Sending Relay ispossibly parsed out of the Received: headers.

• IMPROVEMENT (Secure Messaging add-on): CanIt displays both an INBOX (for receivedsecure messages) and a Sent box (for sent ones.)

• MINOR NEW FEATURE: You can configure the Administration : Show Queue display to hidemessage subjects by default, revealing them only upon clicking a link.

• POLICY CHANGE: IP-based rate-limiting rules are applied first based on the sending relay,and if no rule is found, then based on the connecting relay.

• POLICY CHANGE: When an incident is released, we hold on to any locally-held messagerather than deleting it immediately. This permits the display of a correctly-formatted MIMEmessage in the quarantine display.

• BUG FIX (Secure Messaging add-on): The encryption module could deadlock on very largemessages. This has been fixed.


263

• BUG FIX: If an incident is created in tag-only mode, then the log-indexer links to the incidentpage when displaying the relevant log lines.

• BUG FIX: The “Skip SPF Checks” Known-Networks flag now applies to both the SendingRelay and the Connecting Relay.

• BUG FIX: The Web interface could fail on very old versions of PostgreSQL that lack the “stan-dard conforming strings” parameter. This has been fixed.

• BUG FIX: Setup : Templates includes a template for translating “Country”

• BUG FIX (CanIt-Domain-PRO only): The Stream Count by Realm report could fail on olderversions of PostgreSQL. This has been fixed.

• BUG FIX: A bug in the Autotask billing module could cause the cron job to generate gigabytesof warning messages. This has been fixed.

• BUG FIX: Setup : Domain Overview now shows all domains that have been set up within arealm, even if they lack explicit realm mappings.


• MAJOR NEW FEATURE: A Secure Messaging add-on is available. This lets you create poli-cies for storing mail locally rather than delivering it, and requiring recipients to log on overHTTPS to securely view their messages.

• NEW FEATURE (Hosted CanIt and CanIt-Domain-PRO only): Hosted CanIt and CanIt-Domain-PRO can integrate with Autotask(TM) to automate monthly billing for anti-spam ser-vices.

• NEW FEATURE: You can override scores for SpamAssassin rules. This can be done on aper-stream basis and obeys the normal stream inheritance rules.

• IMPROVEMENT: The “xauth” API call takes an optional “redirect” parameter to automaticallyplace a user in an interior CanIt page after single sign-on.

• IMPROVEMENT: The RSS feed feature has been improved, making it much more useful forkeeping an eye on your quarantine.

• POLICY CHANGE: Set default Storage Manager client operation timeout to 90 seconds insteadof 20. 20 was too short for installations with slow networks and large messages.

• PERFORMANCE IMPROVEMENT (Archiver only): The cron job that expires old archivedmail does it in smaller chunks each night to avoid a very long-running cron job.

• PERFORMANCE IMPROVEMENT: The Storage Manager client/server protocol contains amechanism to reduce network traffic when archiving many copies of the same message.

• DOCUMENTATION IMPROVEMENT: Clarify the distinction between “Connecting Relay”and “Sending Relay” in the context of Known Networks flag “Parse Received Headers”.



• BUG FIX: Make Known Networks “Skip SPF Checks” flag apply to both Connecting Relayand Sending Relay.

• BUG FIX: You can toggle the “Is root?” flag for CanIt users from within the Web interface.Before, you had to delete and recreate the user.

• BUG FIX: Character-set decoding problems in the message preview have been fixed.

• BUG FIX: The Web interface could fail with newer versions of PostgreSQL if stan-dard conforming strings was enabled. This has been fixed.

• BUG FIX: If you have a new-enough version of PHP, the LDAP timeout setting is honored forauthentication attempts. This requires PHP 5.3 or later.

• BUG FIX: If CanIt creates an incident for a tagged message, the log indexer now links to theincident details page.

• BUG FIX: If you used Parse Received Headers with outbound rate-limiting, CanIt could use thewrong IP address to look up rate-limiting rules. This has been fixed.


• NEW FEATURE: On a per-stream basis, the entries in the Pending Notification email can op-tionally be sorted by score ascending rather than date descending.

• NEW FEATURE: All Periodic Reports produce CSV attachments for each chart as well as PDF.This permits the report data to be imported into a spreadsheet and manipulated as required.

• NEW FEATURE: You can create periodic reports showing email address usage.

• MINOR IMPROVEMENT: LDAP user-lookups can force the use of SSLv3 when streaming.

• BUG FIX: The German language localization was completely broken; it is now fixed.

• BUG FIX: Localized column names in reports sometimes had entities double-escaped resultingin things like “ö” appearing instead of the proper character.

• BUG FIX: Some PHP “Strict Standards” warnings were fixed.

• BUG FIX: The wrong permission was being used to control access to the “Statistics” menuentry. This has been fixed.


• BUG FIX: On busy systems, the default “backlog” parameter to listen() for the Storage Managerdaemon may result in errors writing to the storage manager. We have increased the defaultbacklog from 5 to 16 and made it configurable in canit.conf


265

• BUG FIX: If you have enabled the global setting ”Store both raw and decoded messages in inci-dent database” and are using Storage Manager, certain messages could cause protocol violationswith Storage Manager and cause mail to be tempfailed. If your mail logs show many instancesof “PROTOCOL ERROR” from canit-storage-manager, you must upgrade to 9.0.13.


• NEW FEATURE (CanIt-Domain-PRO only): There is a new API call that permits you to re-name a realm.

• NEW FEATURE: The training link template now lets you add templates for whitelist-ing/blacklisting senders. Note, however, that whitelisting or blacklisting a sender always re-quires authentication even if normal voting does not.

• NEW FEATURE (CanIt-Domain-PRO only): The site administrator can completely suspendservice to a realm. This blocks all login attempts and rejects all mail for the realm.

• NEW FEATURE: A new “xauth” API call permits you to create single sign-on links from withinother Web portals.

• NEW FEATURE: In addition to per-sender and per-IP rate-limits, you can also apply per-domain rate-limits to outbound mail.

• NEW FEATURE (Archiver only): There are now API calls to configure archiving.

• UPDATE: ClamAV has been updated to version 0.98

• DEPRECATIONS: Debian 4.0 (“etch”) is no longer supported. Debian 5.0 (“lenny”) is nowdeprecated.

• MINOR CHANGE: The GET /api/2.0/info API call includes more information.

• MINOR BUG FIX: The bulk-entry page would not permit domain-rule entries of the form:.example.com. This has been fixed.

• BUG FIX: LDAP user-lookups would fail mysteriously if an LDAP URL was entered in upper-case: LDAPS://SERVER.EXAMPLE.COM. This has been fixed.

• BUG FIX: CanIt’s “alias” mechanism was case-sensitive. This has been fixed.

• BUG FIX: Several PHP warnings (Strict Mode warnings) have been fixed.

• BUG FIX: The X-Spam-Flag: YES header is correctly added if a message is tagged because ofa filename extension rule.




• POLICY CHANGE: If you make a specific SPF rule for a domain “example.com” and set the“fail” and “softfail” scores to zero, then CanIt does respect domain and sender whitelists for thatdomain, even if SPF fails or softfails.

• BUG FIX: Custom rules and compound rules still did not work correctly for non-latin charactersand certain regular expressions in some cases. This has been fixed.

• BUG FIX (Archiver only): If a message fails to be archived, it is tempfailed so there’s nopossibility of losing messages.


• NEW FEATURE: Add a $Config setting to hide message bodies from users if they are not intheir home stream. This can be used to limit what helpdesk users can see in others’ streams.

• COSMETIC IMPROVEMENT: The License Key page makes it much clearer when a new li-cense key is accepted.

• IMPROVEMENT: Anywhere CanIt asks for an email address to which to send email, you cansupply a comma-separated list of addresses and the mail will go to all of them.

• PERFORMANCE IMPROVEMENT: If a recipient is specified in a log-search, CanIt automat-ically restricts the stream (and realm, in Domain-PRO) to that of the recipient. If this is notdesired, a “contains” relation will prevent the automatic stream restriction.

• MINOR IMPROVEMENT: The page for testing LDAP lookups optionally dumps the entireLDAP entry as LDIF for debugging purposes.

• BUG FIX: Custom rule and compound rule evaluators convert all text lines to Unicode beforeapplying rules. This means that custom and compound rules with non-Latin characters will nowwork correctly on all messages.

• BUG FIX: If you use the “create incidents for tagged messages” feature, CanIt would show thestatus as “Auto-Rejected”. That has been fixed; it now reads “Tagged”.

• BUG FIX: On recent Linux systems such as Debian 7.0, RPTN downloads would fail with an“Unable to validate SSL certificate” error. This has been fixed.

• BUG FIX: Custom and Compound rules use Perl rather than PHP to validate regular expres-sions, ensuring that the full power of Perl regexes is available.

• BUG FIX: The “See Active Queries” link on the Cluster Management page connects as thePostgreSQL super-user to show more information about active queries.

• BUG FIX: A rare edge-case interaction between an unfrozen incident and a whitelist couldallow large spam runs to leak through. This has been fixed.


267

• BUG FIX: CanIt’s SummarizeStatistics task in 9.0.9 was broken on older versions of Perl(5.8.8); this has been fixed.

• BUG FIX: The CanIt Failover SNMP module has been made more forgiving so as not to raisespurious alerts about failover problems.

• BUG FIX (CanIt-Archiver): For journalled messages, CanIt adds all envelope recipients to thelist of addresses for archiving.


• NEW FEATURE: (CanIt-Appliance only): CanIt can be configured to send an email notificationif mail for a given domain starts to queue.

• MINOR IMPROVEMENT (CanIt-Domain-PRO only): The /provision and /domain route APIcalls take extra parameters to configure queued-mail notification (the feature mentioned above.)

• MINOR IMPROVEMENT (CanIt-Domain-PRO only): The address-count-by-realm report nowalso reports the stream count by realm.

• MINOR IMPROVEMENT: A new report charts the number of addresses and streams seen overtime.

• MINOR IMPROVEMENT: The color-selection code has been tweaked so that graphical reportshave more pleasing colors.

• MINOR IMPROVEMENT (CanIt-Appliance only): The System Check task that checks forsufficient free disk space now also checks for sufficient free inodes.

• POLICY CHANGE: In certain specific situations, a tempfail from a verification server is con-verted to a permanent reject. This occurs if the SMTP response from the back-end verificationserver ends with either of the following:

Recipient address rejected: User unknown in local recipient table Mailbox size limit exceeded

• BUG FIX: The “restart-gracefully” and “stop-gracefully” arguments to /etc/init.d/canit-systemdid not correctly stop the CanIt Daemon (canitd) process. This has been fixed.

• BUG FIX: (Archiver only): Archiving rules are now correctly applied to journalled messages.(Before, journalled messages were archived unconditionally.)

• BUG FIX: The code that counts the number of email addresses seen now correctly removesBATV tags and sendmail “plus-hack” suffixes to avoid multiply-counting the same email ad-dress.

• BUG FIX: Domain-routing entries that start with a dot could cause the ticker system-check taskto die. This has been fixed.

• BUG FIX: Theme customization would not work on versions of PostgreSQL older than 8.2; thishas been fixed.



• BUG FIX (CanIt-Domain-PRO only): The system prevents you from deleting a realm-mappingif a domain-routing entry exists for the domain.


• UPGRADES: Many included Perl modules have been updated to more recent versions.

• EXPERIMENTAL: We now have Debian 7 “Wheezy” packages and ISO images. These arestill considered experimental.

• NEW FEATURE: IMAP and POP3 user-lookups can rewrite the login name to a stream usinga Rewrite Expression.

• NEW FEATURE: CanIt stores and displays the “full name” from the From: header in the quar-antine display and the archive display (archiver add-on only.)

• POLICY CHANGE: By default, normal users cannot reopen incidents. Only administratorscan.

• MINOR IMPROVEMENT (Appliances Only): The log search feature allows you to search byminimum/maximum score, “reason” and “detail” fields. (This applies only to log lines indexedafter the 9.0.8 upgrade.)

• MINOR IMPROVEMENT: The Web interface can be put into “Maintenance Mode”. This pre-vents users from doing anything and displays a maintenance notice of your choice.

• MINOR IMPROVEMENT: More theme elements are customizable.

• MINOR IMPROVEMENT: You can choose to place the action buttons on the left or the rightin the quarantine display.

• MINOR IMPROVEMENT: A “danger” sign warns of hazardous attachments like EXE files tomake it clear that a quarantined message might be malware.

• MINOR IMPROVEMENT: The replacement sequence “%=X” is replaced with the number ofX’s equal to the spam score, where X can be any character except “%”.

• BUG FIX: Fixed race condition that could result in tempfail when attempting to create an inci-dent.

• BUG FIX: Fixed various edge-cases in the ”Create incidents for tagged messages” feature.


• MAJOR NEW FEATURE: If CanIt has enough local Bayes statistics for a given token, it can uselocal statistics in preference to inherited statistics and RPTN. This means that CanIt respondsmore quickly to training. Our experience with this feature is limited, so by default it is disabled.Turn it on under Preferences : Quarantine Settings by setting S-2410 to “Yes”. You may noticehigher false-positives for a while until sufficient local training has been created.


269

• MINOR IMPROVEMENT: You can use the %{empty} tag in tag-only mode to prevent thesubject from being tagged (but the X-Spam-Flag: YES header is still added if appropriate.)

• UPGRADE: ClamAV has been upgraded from 0.97.7 to 0.97.8.

• PERFORMANCE IMPROVEMENT: If you are using Storage Manager, the code that createsan incident keeps a database transaction open for less time than before.

• BUG FIX: If you create incidents even in tag-only mode, CanIt could end up auto-rejecting thesecond and subsequent copies of the same email. This has been fixed.

• BUG FIX (CanIt-Domain-PRO appliance only): Messages that were forced into a stream couldhave their log lines indexed in the wrong realm. This has been fixed.

• BUG FIX: Several deprecated Perl constructs have been fixed.

• BUG FIX (CanIt-Domain-PRO only): In a highly-unlikely edge case, CanIt-Domain-PRO coulduse an incorrect user-lookup for a domain. This has been fixed.

• BUG FIX: The Regular Expression web page uses a Perl helper script to test the regular expres-sions, which gives more accurate results than doing it in PHP.

• BUG FIX: Many PHP constructs that yielded errors in PHP 5.4 have been fixed.

• BUG FIX: If a realm has domain-related information like domain mappings or authenticationmappings, the GUI and API prevent remapping the domain unless the related mappings are firstdeleted.


• POLICY CHANGE: Doing a quarantine search by sender searches only the header sender; thequery to search envelope senders could be pathologically slow in some situations.

• MINOR NEW FEATURE: The Cluster Management page lets the site administrator display asnapshot of active database queries.

• MINOR IMPROVEMENT: A sample “site failover” script has been included that notifies ad-ministrators of an impending notification and lets them either force or cancel failover.

• PERFORMANCE IMPROVEMENT (CanIt-Domain-PRO only): The query to locate an exist-ing incident based on its hash has been made faster.

• BUG FIX: The RSS Feed menu item would be displayed even if a user lacked RSS permission(though the menu entry would display Permission Denied if clicked.) Now the menu entry iscorrectly hidden.

• BUG FIX: In certain circumstances, CanIt could fail to send notifications to people with pendingmessages in the quarantine. This bug was introduced in release 9.0.1, so anyone running 9.0.1through 9.0.5 should upgrade.




• NEW FEATURE: In a CanIt cluster, cluster members can be grouped by location. When CanItsynchronizes Bayes data across a cluster, it attempts to minimize bandwidth used to copy filesfrom one location to another. This improves performance if your cluster has several groups ofmachines connected by relatively low-bandwidth links.

• NEW PERMISSION (CanIt-Domain-PRO only): You can disable the ability of realm adminis-trators to see/edit User Lookups. This may be important if they contain passwords that shouldnot be revealed to subrealm administrators.

• NEW REPORTS: CanIt now features a report showing the top Operating Systems seen forrecent messages. This works only if your platform supports the Passive OS Fingerprintingmodule.

• POLICY CHANGE: In tag-only mode, if a message is tagged as spam, the headers “Precedence:bulk”, “X-Auto-Response-Suppress: All” and “Auto-Submitted: x-no-autoresponse-please” areadded. This is designed to prevent out-of-office software from auto-responding to tagged mail.

• PERFORMANCE IMPROVEMENT: Failover initialization performs less data copying anduses less disk space than before.

• BUG FIX: The failover system now operates correctly on databases that use multiple ta-blespaces.

• BUG FIX (CanIt-Domain-PRO only): The system now prevents you from deleting a realm-mapping if domain-related information such as Verification Servers, Domain Mappings or Au-thentication Mappings exist for domains within the realm mapping.

• BUG FIX: The system now prevents you from marking “default” as a special stream.

• BUG FIX: A bug that prevented theme customization in themes with more than one customiz-able image has been fixed.

• BUG FIX: If a user has access to all streams (by virtue of having “*” in the Accessible Streamslist), stream auto-complete now works.

• BUG FIX: During outbound rate-limiting, CanIt would sometimes send multiple notificationsdue to a race condition. Now it only sends exactly one notification.


• NEW FEATURE: The RPM and Appliance versions of CanIt include passive OS fingerprintingwhich attempts to guess the operating system and link type of the SMTP client. The results aretokenized for Bayes and can be used in Compound Rules.

• NEW FEATURE: The API permits a POST to /rules to bulk-create rules. This can be substan-tially faster than iterating on the client side.


271

• UPDATE: We ship ClamAV 0.97.7 (updated from 0.97.6)

• GUI IMPROVEMENT: The window that pops up in response to clicking on a voting link makesit clearer that the vote has taken place and no further action is required.

• GUI IMPROVEMENT: Emblems in the Quarantine Display show SPF pass/fail/softfail results.

• IMPROVEMENT: We store and display both the first and last occurrence of an anomaly (underAdministration : Anomalies) rather than just the last occurrence.

• IMPROVEMENT: The Compound Rule compiler has been split out from the run-time evalua-tion of compound rules to save memory.

• IMPROVEMENT: The internal “Circuit Breaker” code that backs off from contacting deadservers remembers the last error message and uses it for more informative anomaly messages.

• IMPROVEMENT: The comments (if any) associated with Compound Rules and Custom Rulesare added to the spam report.

• IMPROVEMENT: Various sections of the Pending Notification have class attributes, allowingthem to be suppressed with CSS code in the template “Header for ’Webform’-style PendingNotification”.

• IMPROVEMENT (CanIt-Domain-PRO appliance only): The log indexer tries much harder todetermine which realm various log messages apply to, meaning realm administrators have ac-cess to more of their logs than before.

• POLICY CHANGE: The “Email Address Usage” reports only consider email addresses seenwithin the last 30 days.

• POLICY CHANGE: The default timeout for LDAP lookups has been reduced from 120 secondsto 20 seconds.

• POLICY CHANGE: Sender blacklists created in response to clicking “Blacklist Sender” in aPending Notification now expire after 60 days rather than persisting forever. This policy changewas made after determining that most such blacklists were not useful because they blacklisteddisposable addresses.

• POLICY CHANGE: The “Silently Discard” option for handling incidents has been removed.In most deployment scenarios, “Reject Message” is actually the same as “Silently Discard”anyway.

• POLICY CHANGE: The minimum counts before using Bayes data now apply to the aggregateof all Bayes databases rather than each Bayes database individually. This makes CanIt startusing personal Bayes data much sooner than before.

• BUG FIX: An explicit “Reject” for a sender or domain now overrides a “Hold if looks likespam” rule from a parent stream.

• BUG FIX: The Quarantine Search would fail to find messages with accented (or any non-ASCII)characters in the subject. This has been fixed.



• BUG FIX: The Bayes @@PARENTS inheritance mechanism has been completely fixed. Be-fore, the Web interface would display misleading information.

• BUG FIX: Minor problems with the Compound Rule entry page have been fixed. It is also nowpossible to see compound rules in all streams if you switch to the “*” pseudo-stream.

• BUG FIX: The API would not permit creation of a “*” address mapping; this has been fixed.

• BUG FIX: The API would not permit a domain rule of the form “.example.com” with a leadingdot. This has been fixed.

• BUG FIX: An error in the compound rule compiler has been fixed. Upon upgrade, all compoundrules will be recompiled to ensure that the generated code is correct.


• NEW FEATURE: You can add your own custom X- header to delivered messages. The headertemplate includes several substitution tags that will be replaced on delivery; see the User’s Guidefor details.

• IMPROVEMENT: The Bulk Entry rule page lets you set an expiry date on rules you enter.

• IMPROVEMENT: Compound Rules can now use the SPF and DKIM results as part of the rule.

• IMPROVEMENT (CanIt-Domain-PRO only): Under Setup : Templates in a non-base realm,you can revert a template to its inherited value.

• IMPROVEMENT: The “Viewing Stream” display has been improved to more clearly showstream inheritance.

• IMPROVEMENT: Pending Notifications now include the country-code of the SMTP relay andan indication if the envelope sender differs from the From: header.

• BUG FIX: In Rules : Sender, you can select “Tempfail” in the filter action.

• BUG FIX: The “Simplified Interface” ignored any logo customization (under Setup : ThemeCustomization.) This has been fixed.

• BUG FIX: In the Compound Rules editor, pressing Enter in the comment field would result inthe “Delete” button being activated instead of “Save”. This has been fixed.

• BUG FIX (CanIt-Domain-PRO only): The mechanism to propagate Bayes hand-votes up therealm hierarchy did not work correctly. This has been fixed.


• POLICY CHANGE: Due to significant demand, we have reinstated the “Clickable Webform”pending notification type.


273

• POLICY CHANGE: You can now select whether Pending Notification emails only notify aboutnew incidents created since the previous notification (9.0.1 behavior) or all pending incidents(pre-9.0.1 behavior.) The default is to notify only about new incidents since the previous notifi-cation.

• NEW FEATURE: You can add arbitrary CSS when you customize a theme, allowing for veryfine control over the theme’s appearance.

• MINOR NEW FEATURE (CanIt-Domain-PRO only): You can specify @@PARENTSn (whenn is a decimal number) to inherit Bayes training from a limited number of ancestor streams.

• BUG FIX: The @@PARENTS Bayes-training inheritance setting was not accepted by the GUI.This has been fixed.


• POLICY CHANGE: Please read the CanIt license (found in an appendix in the AdministrationGuide and User’s Guide) for a disclaimer about time-critical mass-mailings.

• POLICY CHANGE (CanIt-Domain-PRO only): Realm administrators can delete realm map-pings they own, effectively deprovisioning a domain.

• POLICY CHANGE: The “Clickable Webform” pending notification type has been removed. Itdidn’t work in most email readers anyway.

• POLICY CHANGE: Pending Notification emails now only include any new incidents createdsince the previous notification.

• MAJOR NEW FEATURE: You can create compound custom rules. These let you combineconditions with boolean operators. Because of the power of these rules, they are available bydefault only to administrators, although end-users can be granted permission to create them.

• MAJOR NEW FEATURE: Rate-limiting rules are much more versatile, permitting per-sendersettings and including the ability to hold all mail from a rate-limited sender.

• NEW FEATURE (CanIt-Domain-PRO only): A new “provision” API call has been added tosimplify provisioning new domains. The site administrator can grant provision permission totrusted realm administrators.

• NEW FEATURE: An aliasing feature has been added; this causes CanIt to actually rewriterecipient addresses (as opposed to simply streaming them into one stream but not rewriting thedestination address.)

• NEW FEATURE (Appliance only): CanIt allows administrators to temporarily pause deliveryto specific domains. This can be useful if a back-end mail server is undergoing scheduledmaintenance.

• NEW FEATURE: CanIt can cache login credentials so that if a back-end IMAP, POP3 or LDAPserver goes down, users can still log in (providing they have successfully logged in recently.)



• NEW FEATURE: You can request CanIt to create incidents even in tag-only mode. This maymake it easier to determine why a message was tagged.

• NEW FEATURE: The “Strip Attachment” feature now lets you specify that administrative per-mission is required to release attachments that were stripped and held on the server.

• NEW FEATURE: The “See Mail Queue” page lets you summarize queued messages by desti-nation domain.

• NEW FEATURE: Hand-votes in a stream are also recorded in the “default” stream (and inthe case of CanIt-Domain-PRO, the “default” stream of all ancestor realms.) This effectivelycreates site-wide Bayes databases which can be used by specifying that streams should usetraining from @@PARENTS (see manual for details.)

• NEW FEATURE (Archiver only): You can create rules that specify whether or not to archivemessages. This lets you avoid archiving machine-generated notifications or other messages thatyou don’t want archived.

• NEW API CALL: GET /api/2.0/address to stream/[email protected] returns the stream towhich “[email protected]” would be mapped.

• GUI IMPROVEMENT: Several icons in the Quarantine Overview page display additional in-formation about incidents such as SPF fail/softfail and important notes about the incident.

• IMPROVEMENT: Several new SNMP variables to monitor the health of the CanIt server arenow available: Total number of system checks, number of failed system checks, and total num-ber of anomalies detected.

• MAJOR IMPROVEMENT: RPTN reporting has been revamped to try to preserve as manyhand-votes as possible rather than dropping excessive votes.

• MAJOR IMPROVEMENT: CanIt detects dead back-end servers and refrains from using themfor a short period of time. This can help mitigate load problems if back-end servers disappearoff the network.

• CHANGE: The notification setting ”Add ’Blacklist Sender/Whitelist Sender’ Links to HTMLNotification” has been split into two independent settings (one for adding whitelist links andanother for adding blacklist links.) On upgrade, the existing setting is migrated to both the “Addwhitelist links” setting and the “Add blacklist links” setting.

• MINOR NEW FEATURE: A “Domain Overview” page summarizes important per-domain set-tings and is useful for troubleshooting.

• SECURITY FEATURE: You can configure the CanIt web interface to lock sessions to a singleIP address or a small range of IP addresses. This can help thwart session-hijacking attacks.

• GUI IMPROVEMENT: The Known Networks page has a filter box that lets you reduce clutterif you have many Known Networks entries.

• MINOR GUI IMPROVEMENT: “Alternate Addresses” has been renamed to ”My Addresses”


275

• MINOR GUI IMPROVEMENT: “Stream Settings” has been renamed to “Quarantine Settings”

• MINOR GUI IMPROVEMENT: “Address Mappings” has been renamed to “Address-to-StreamMappings”

• MINOR IMPROVEMENT: The LDAP user-lookup wizard now has pre-canned settings forallowing users to log on with their email address or their Active Directory username.

• MINOR IMPROVEMENT: You can decide whether to ignore whitelists for SPF “fail” or “soft-fail” with two separate settings instead of one setting that applied to both fail and softfail.

• MINOR NEW FEATURE: The LDAP user-lookup code can help determine the form of theActive Directory Bind DN setting. This makes setting up Active Directory much easier.

• MINOR IMPROVEMENT: Weekends are shaded on most reports in the Web interface.

• MINOR IMPROVEMENT: An emergency mechanism to disable theme customization has beenadded to recover from customization mistakes that make the GUI unreadable.

• MINOR IMPROVEMENT: The Perl CanIt::API::Client module has a get last value() method;see the man page for details.

• BUG FIX: Searching the trap used case-sensitive matching for “is” relationships, which couldyield incorrect search results. This has been fixed.

• BUG FIX: The %u and %d sequences in LDAP searches did not work for authentication. Thishas been fixed.

• BUG FIX: CanIt would fail to compile with FreeBSD’s make program. This has been fixed.

• BUG FIX: The Storage Manager creates a lock file to ensure that there will never be two back-ground pruning processes running at the same time. This would never have caused an error, butdid cause excessive disk I/O.

• BUG FIX: The canit-failover-setup.pl would create invalid authorized keys files. This has beenfixed.

• BUG FIX: A rare edge-case could make data expire out of Storage Manager before it expiredout of the database, causing message retrieval to fail. This has been fixed.

• BUG FIX: A forced-to-stream incident could be re-trapped in a different stream after it is re-leased. This has been fixed.

• BUG FIX (Appliances only): The log-indexer did not correctly index all recipients if a line waslogged with more than one recipient. This has been fixed.

• BUG FIX (Archiver only): A message with a missing charset=xxx parameter could be displayedincorrectly. This has been fixed.




• NEW FEATURE: CanIt sets the “canit user” note for Apache. You can use this to log CanItusers to your Apache log with the log format sequence %{canit user}n in the LogFormat direc-tive. On our Debian-based appliances, we adjust the Apache configuration file to do this.

• NEW FEATURE: CanIt’s RP-Web theme features a “Mobile” view optimized for mobile de-vices. This is still considered experimental; feedback is welcomed.

• NEW FEATURE: The nightly Storage Manager maintenance task is extremely I/O intensive.You can set a configuration setting in canit.conf to have Storage Manager limit its use of diskbandwidth (at the expense of taking longer to run the nightly maintenance task.)

• NEW FEATURE: The failover system supports Streaming Replication on versions of Postgresql>= 9.0. See the Cluster Guide for details.

• MINOR NEW FEATURE (Archiver): A new “to or from” search criterion has been added as aconvenience. (It saves entering two separate criteria.)

• PERFORMANCE IMPROVEMENT: You can parallelize the releasing and remailing of quar-antined messages. See the Administration Guide for details.

• IMPROVEMENT: Storage Manager uses fewer levels of subdirectories for storing data. Thisshould reduce inode consumption and slightly improve performance.

• MINOR IMPROVEMENT: If you view the mail queue sorted by domain, CanIt displays thecount of queued messages for each domain.

• MINOR IMPROVEMENT: The “Automatically Populate Notification Address” setting isshown under Preferences : Notifications to administrators in the default stream.

• BUG FIX: CanIt’s notification messages could not handle subjects or From: headers with non-ASCII characters. This has been fixed; CanIt can handle any UTF-8 character sequence now.

• BUG FIX: In older versions, it was not possible to create a Rewrite User-Lookup using the API.This has been fixed.


• MAJOR NEW FEATURE: The Web interface colors and logos can be themed from within theWeb interface itself. In CanIt-Domain-PRO, realm administrators can adjust the colors andlogos seen by their realms and subrealms.

• PERFORMANCE IMPROVEMENT: The nightly cron job adds additional indexes to the logindexes, making it faster to search by subject, sender or message-ID.

• IMPROVEMENT: Additional helper scripts make setting up failover and recovering after afailover much easier and less error-prone.


277

• IMPROVEMENT: You can reset stream-setting inheritance on a per-setting basis. Before, youcould only reset it for all settings in a stream at once.

• IMPROVEMENT: All of the manuals are now available as online HTML as well as PDF.

• IMPROVEMENT: Various tests (such as Blacklisted Recipients and Verification Servers) logmore details about why a recipient address is rejected, making it easier to diagnose deliveryproblems.

• MINOR NEW FEATURE: You can reset open incidents to “Pending” if you make a mistakeand want more time to consider how to dispose of an incident.

• POLICY CHANGE: New installations receive a sensible default SPF rule. On upgrade, oldinstallations will receive the same rule if they currently lack a default SPF rule.

• POLICY CHANGE: The maximum size of messages to scan for spam has been increased from150kB to 1MB. There are some large spams out there...

• POLICY CHANGE: The “Catch Rate” slider has been removed from the Integrated Interface.It was not useful and could be misleading.

• BUG FIX (CanIt-Domain-PRO only): The “Daily Mail by Realm” background report couldimproperly leak cross-realm information; this has been fixed.

• BUG FIX: The failover code that makes a base backup could fail if the initial rsync took a verylong time. This has been fixed.

• BUG FIX: A very rare edge-case problem with parsing received headers and avoiding whitelistson SPF failure has been fixed.

• BUG FIX: More cron-job failures raise a system check warning. In the past, some logged errorsbut were otherwise silent.

• BUG FIX: Add-on product keys are checked for expiry and used to raise system check warnings.

• BUG FIX: Code that depended on a newer version of PHP than ships with Red Hat has beenfixed.

• BUG FIX: If a Bayes signature has expired, the Vote page provides a more useful error message.

• BUG FIX: The canit-api-client tool did not accept the –port command for “domain route up-date”. NOTE: You need to run “canit-api-client introspection clear” after upgrading for thechange to take effect.


• IMPROVEMENT: If you use CanIt’s web interface over HTTPS, the CANIT cookie has the“secure” attribute set.



• MINOR NEW FEATURE: The startup script /etc/init.d/canit-system can stop CanIt “grace-fully”. This means that it waits for any processes doing critical work that should not be inter-rupted to exit on their own rather than forcibly killing them.

• MINOR NEW FEATURE: A new canit.conf setting allows you to have all MIMEDefang filesand sockets be group-accessible.

• MINOR NEW FEATURE: The time-span over which to track email addresses that have beenseen is configurable (rather than being hard-coded at 61 days.)

• MINOR IMPROVEMENT: The SMTP reply generated when a message scores over the auto-reject-and-do-not-create-incident threshold is now templatable. (It used to be hard-coded.)

• POLICY CHANGE: CanIt-generated notices use the template “Source E-Mail address of CanItnotifications” as the envelope header instead of <>. To go back to the old behavior, set thetemplate to “<>”.

• POLICY CHANGE: Subdomain-expansion for DKIM and SPF rules has changed. Check theUser’s Guide for details.


Please *carefully* read the SPF and DKIM sections of the User’s Guide to see how subdomainexpansion works in this release.

• POLICY CHANGE: The thresholds in the failover SNMP module have been made more real-istic... the old values would often trigger false alerts about problems with failover.

• PERFORMANCE IMPROVEMENT: You can parallelize sending of Pending Notifications. Seethe Administration Guide for details.

• PERFORMANCE IMPROVEMENT: The nightly check to look for domains that don’t validaterecipients can be run in parallel. See the Administration Guide for details.

• PERFORMANCE IMPROVEMENT: The code to check for an existing incident has aworkaround for a potentially-poor PostgreSQL query plan.

• API IMPROVEMENT: The API call: GET /api/2.0/realm/@@/stream/STREAMNAME func-tion now includes an “active” member in the returned hash. This is set to 1 if the stream is inthe Active Streams list and 0 otherwise.

• BUG FIX: The feature “Insert Streamed Mail Directly Into Sendmail Queue” did not lock queuefiles correctly, which could lead to damaged mail. This bug has been fixed and the feature isnow safe to use.

• BUG FIX: If the backup database server uses pgbouncer, failover could fail. This has beenfixed.

• BUG FIX (Debian appliances only): A typo that broke auto-updates has been fixed.

• BUG FIX: After adding or deleting a known network, the known networks page would some-times display incorrect information. This has been fixed.


279

• BUG FIX: The documentation notes (and the GUI enforces) that the ticker host must also be aninbound scanner.

• BUG FIX: The Permissions system did not correctly restrict viewing of the Preferences : Noti-fications page. This has been fixed.


• MAJOR CHANGE (Appliance Only): The log-indexing feature now uses PostgreSQL insteadof Xapian. Also, the log-searching API calls have changed; be sure to read the API guidecarefully.


After you upgrade from 8.1.0 to 8.2.0, all your logs will be re-indexed, which may be slow. Ifyou run a busy CanIt system that uses log-indexing, please contact Roaring Penguin supportbefore upgrading to 8.2.0 so we may plan capacity for the log-indexer.

• IMPROVEMENT: On our Debian appliances, we have added a script that makes it easier to setup database failover. This script eliminates many error-prone and tedious configuration steps.

• GUI CHANGE: We have revamped the look of CanIt with a new “RP-Web” theme, which isnow the default. You can always change back to the previous “Postmodern” theme if you preferit.

• NEW FEATURE (Appliance Only): You can specify a non-standard port (ie, other than 25) formail routing.

• NEW FEATURE: The IMAP, POP3 and LDAP user-lookups optionally allow you to force theusername to lower-case.

• NEW FEATURE: Administrators (and realm administrators in CanIt-Domain-PRO) can switchusers to any other user. This allows them to see the interface exactly as it would be seen byother users.

• NEW FEATURE: You can ask not to be notified of pending messages that score above a speci-fied threshold. This can reduce the size of notification messages by not notifying you of obviousspam.

• DOCUMENTATION IMPROVEMENT: The online manuals have several embedded tutorialvideos.

• IMPROVEMENT: A couple more standard auto-replies are recognized to avoid auto-whitelisting on an auto-reply.

• IMPROVEMENT (Appliance Only): Several more types of log lines are recognized and givenexplanations.



• POLICY CHANGE: An SPF, DKIM or Domain rule on “example.com” would *also* apply toall subdomains of “example.com”. This is no longer the case. However, you can make a ruleon “.example.com” (note the leading dot) that applies only to all subdomains of example.com.Consult the manuals for details.

• POLICY CHANGE: The “Mismatch Rule” feature was removed. It was easily abused and hasbeen obsoleted by SPF.

• POLICY CHANGE: We no longer ignore whitelists if the Header From: sender would fail SPFtests. The previous policy broke many mailing lists.

• POLICY CHANGE: The Anomaly system was too sensitive. We now do not report on mostanomalies unless they happen 30 times or more.

• GUI CHANGE: The “Trap Contents” menu item was renamed “Quarantine”; the latter termi-nology is more common.

• IMPROVEMENT: The mail queue display can be sorted by the domain of the (first) recipient.

• IMPROVEMENT (API): We now offer full API-level access to Known Networks.

• IMPROVEMENT (API): All PUT API calls now have corresponding POST calls that updateexisting resources or create new ones.

• IMPROVEMENT: The system warns if you whitelist a domain that lacks an SPF record.

• IMPROVEMENT: A user marked read-only can make absolutely *no* changes to the system,even if he/she has administrative access. Read-only users are therefore far more useful forhelpdesk personnel than they were previously.

• BUG FIX: The rate-limiting feature sometimes would not kick in immediately when the ratelimit was exceeded. This has been fixed.

• BUG FIX: Edge-cases that could make Verification Server checks fail have been fixed.

• BUG FIX: A rare edge-case that could lose requests to remail held messages if a Storage Man-ager node was restarted has been fixed.

• BUG FIX: If an incident is created in a special stream, we log that stream as well as the streamused for rules.

• BUG FIX: Some reports treated sender and recipient addresses case-sensitively resulting inincorrect statistics. This has been fixed.

• BUG FIX (Domain-PRO only): When a realm was deleted, its children were incorrectly repar-ented under “base” instead of the deleted realm’s parent. This has been fixed.


281


• MAJOR IMPROVEMENT: Bayes tokens are now stored in Unicode. This makes CanIt farmore effective than before on non-Latin character sets. We strongly recommend that all CanItusers upgrade just to get this feature.

• IMPROVEMENT: It is easier to add and remove scanner nodes to and from the cluster “onthe fly”. This allows you to add scanners during busy periods and remove them to save powerduring quiet periods.

• NEW FEATURE: Additional system checks warn if there are too many PostgreSQL WAL filesand if there are any queue files older than 10 days in the Sendmail queue.

• NEW FEATURE (CanIt-Domain-PRO only): You can disable nightly “Anomaly Warning” no-tifications.

• NEW FEATURE: A “hook” mechanism has been implemented allowing custom scripts to runwhen certain events occur. See the file /usr/share/canit/hooks/README for details.

• NEW FEATURE: /etc/init.d/canit-system has a new “stop-most” argument that stops almostall CanIt services, but leaves running any services (such as PgBouncer) that are essential forconnecting to the database.

• POLICY CHANGE: By default, sender and domain whitelists are ignored for messages that failSPF (with a “fail” or “softfail” result.) This makes it much safer to whitelist domains such aspaypal.com or ebay.com without falling victim to phishing attacks because of the whitelist.

• POLICY CHANGE: If a given incident is retransmitted more than 10 times from a given IPaddress, we stop counting the actual number of transmission attempts. Incrementing this countercan cause severe database contention leading to a performance slowdown.

• POLICY CHANGE: On new installations, we default to downloading and submitting RPTNdata.

• IMPROVEMENT (CanIt-Domain-PRO only): Several web pages now allow you to choosebetween displaying items only in the current realm or items in the current realm and all sub-realms.

• NEW API CALL: GET /api/2.0/domain recipient verification returns information aboutwhether or not domains correctly validate recipients and whether or not their MX records pointat the CanIt cluster.

• NEW API CALL: GET /api/2.0/realm/REALMNAME/anomalies returns the list of anomaliesfor REALMNAME and all of its subrealms.

• NEW API CALL: GET /api/2.0/system check returns the list of system checks.

• PERFORMANCE IMPROVEMENT: The Storage Manager client code does not “ping” theserver as often as before, reducing Storage Manager round-trip times.



• PERFORMANCE IMPROVEMENT (CanIt-Domain-PRO only): In several places, caching hasbeen added to reduce database lookups on the realms table.

• DOCUMENTATION IMPROVEMENT: Note that changing a Storage Manager fromRead/Write to Read-Only or vice-versa requires restarting CanIt on that node.

• CHANGE (CanIt-Domain-PRO only): The pending notification subject by default includes thestream name as “stream” rather than “realm:stream”. The old behavior can be reinstated byusing “%{fullstream}” in the template.

• BUG FIX: The code for viewing the mail queue now understands Sendmail queues with separateqf/ and df/ directories.

• BUG FIX: Under certain circumstances, a message that required streaming could be deliveredeven if the streams’ Maximum Message Size parameter was smaller than the actual messagesize. This has been fixed.

• BUG FIX: An error in permission calculation that could sometimes make permissions too stricthas been fixed.

• BUG FIX: If a cluster node’s inbound or outbound flag is changed, Sendmail map files are nowregenerated.


• MAJOR NEW FEATURE: The Bayes tokenizer converts all tokens to Unicode, allowing fortokenization of non-Latin character sets. Note that currently, the non-Latin data is not used bythe Bayes analyzer; it is used merely to feed back to RPTN and build up non-Latin tokens.The next release of CanIt will fully support Unicode tokens and be able to tokenize mail in anylanguage. NOTE: This feature may double the number of .cdb files in the Bayes directories.The double-files will be removed gradually by the next version of CanIt.

• EXPERIMENTAL NEW FEATURE: A scanner-only cluster member may be gracefully re-moved from the cluster (Debian Appliance only.) In this mode, the cluster member stops pro-cessing external mail and shuts down as soon as its queues have been drained.

• NEW FEATURE: The trap display indicates with an icon that a message has attachments. Hov-ering over the icon reveals the attachment names and MIME types.

• NEW FEATURE: Administrators can view the mail queue from within the CanIt web interface.

• NEW FEATURE: The rate-limiting feature allows you to tempfail all mail from a sender or IPaddress instead of rejecting it outright.

• NEW FEATURE: A new “Header From: address of sender of CanIt notifications” lets you setthe From: header address for CanIt notifications.

• NEW FEATURE (Archiver only): The importer script lets you restrict which messages to im-port by date.


283

• NEW API CALL (Domain-PRO only): GET /realm/XXX/subtree returns the subtree of realmsrooted at XXX

• IMPROVEMENT: The “E-Mail address of CanIt System Administrator” may be set to acomma-separated list of addresses. All addresses will receive alerts sent to the system ad-ministrator.

• IMPROVEMENT: The LDAP user-lookup lets you specify a list of attributes to use for thestream name. The first one found is used.

• IMPROVEMENT: The LDAP user-lookup provides additional options for streaming an addresswhose lookup succeeds but that lacks a stream attribute. In addition to streaming to “default”,you can fall back on the AsIs, ChopUser or ChopDomain methods.

• PERFORMANCE FIX (Domain-PRO only): Parts of the Web interface were very slow oninstallations with thousands of realms. This has been fixed.

• POLICY CHANGE: “Subject” custom rules are applied only to the decoded subject. If youwant to apply a rule to the raw subject, use a “Header” regular-expression custom rule.

• POLICY CHANGE: Our Bayes algorithm was modified slightly to thwart common Bayes poi-soning attacks.

• BUG FIX: The PDF background-report formatting could mess up if your report includes manyclassifications. This has been fixed.

• BUG FIX: All short PHP tags <?= ?> have been replaced with proper <?php ?> tags becausesome sites disable short tags.

• BUG FIX (Domain-PRO only): The Audit Trail feature could leak cross-realm information.This has been fixed.

• BUG FIX: Various permission-related API bugs were fixed.

• BUG FIX: For consistency, the GET /realm/XXX/streams API call sets a “parent stream” at-tribute in each returned stream.

• BUG FIX: Reports would show the null sender as an empty string rather than <>. This hasbeen fixed.

• BUG FIX: The “Anomaly Detection” feature introduced in 8.0.12 was too sensitive. It now onlywarns of anomalies that happen more than 30 times in three days.

• BUG FIX: A programming error could make the Storage Manager server enter an infinite loopand consume as much CPU as it could grab. This has been fixed.

• BUG FIX: The failover code would not restart CanIt on the backup database server after afailover. This has been fixed.




• NEW FEATURE: Unexpected problems (for example, errors communicating with LDAPservers or problems with verification servers) are logged in the new Administration : Anomaliespage. If there are any anomalies, administrators (realm administrators in CanIt-Domain-PRO)are emailed nightly.

• NEW FEATURE (Archiver only): You can reply to an archived message from within the CanItweb interface. To prevent abuse, a given user can redeliver or reply to at most three recipientsat a time, and can reply or redeliver only once every 30 seconds.

• NEW FEATURE (Archiver only): You can view the archive as monthly “folders” making thearchiver web interface more webmail-like.

• NEW FEATURE: Filename extension rules are applied to filenames found within zip archivesas well as directly within the email. This feature is always available on CanIt appliances and isavailable on other platforms if the “zipinfo” program is installed.

• EXPERIMENTAL NEW FEATURE: The archiver can import existing mail archives in mboxor “PST” format.

• POLICY CHANGE: When a new member is added to the cluster, it is always assigned thelowest-available host number.

• COSMETIC IMPROVEMENT: The Reports page no longer draws pie charts with an absurdly-huge number of pie slices. It truncates the number at 20 slices by default.

• BUG FIX: The Known Networks page would create invalid Sendmail access table entries forIPv6 networks. This has been fixed.

• BUG FIX (Domain-PRO only): On installations with many realms, some pages would rendervery slowly. This has been fixed.

• BUG FIX (Appliance only): The Sendmail aliases file was incorrectly coded as “/etc/aliases”instead of “/etc/mail/aliases”. This has been fixed.

• BUG FIX: The code to parse Received: headers would sometimes pick out invalid IP addresses.This has been fixed.

• BUG FIX: Invalid envelope addresses would sometimes be parsed incorrectly; this has beenfixed.

• BUG FIX: A very rare race condition in incident creation has been fixed.

• BUG FIX: In the User Lookup wizard, do not allow users to turn off “Use for Streaming” or“Use for Authentication” if a domain is already using the user-lookup for that purpose.

• BUG FIX: Setup : HTTPS was broken by the 8.0.11 release; it is now fixed.

• BUG FIX: The trap display would sometimes produce incorrectly-sorted results when sortingby sender address or domain. This has been fixed.


285

• BUG FIX: The option “Skip RBL Checks” for Network Rules was inadvertently removed de-spite being documented in the manual. This option has been reinstated.

• BUG FIX (CanIt-Domain-PRO only): The “rlm” URL parameter was not handled consistentlyand could lead to odd behavior in the Web interface. This has been fixed.


• NEW PERMISSIONS (CanIt-Domain-PRO only): You can now set read-only and read-writepermissions on realm expiry dates and descriptions.

• NEW FEATURE: Rather than hard-coding the number of messages per pending notification at40, CanIt now allows you to set the limit to any number up to 1000.

• NEW FEATURE: A new preference (“Use a vertically-compact trap display”) makes CanIt tryto use less vertical space in the trap display.

• NEW FEATURE: You can specify different rate-limits for senders as opposed to IP addressesin Known Networks. For example, you can set a limit of 100 recipients/hour for a given senderand 500/hour for a given IP address.

• NEW FEATURE (Archiver only): If internal mail comes from many or unknown IP addresses,you can use a shared secret to authorize CanIt to archive mail as internal or outbound mail.

• POLICY CHANGE: For LDAP streaming methods, the default if no stream attribute is foundis to put the message in the “default” stream rather than tempfail it.

• MINOR FIX: When a sender address is auto-whitelisted, the comment added to the auto-whitelist entry specifies details: (from <SENDER> via <IP>)

• INTERNAL CHANGE: The Perl CanIt::API::Client module uses JSON rather than YAML asthe serialization format. The PHP YAML parser/generator is buggy and the YAML specificationis overly-complex. Using JSON should eliminate many edge-case bugs.

• INTERNAL CHANGE: CanIt uses JSON rather than YAML for communication between clus-ter members. This should have no visible impact other than the elimination of a few edge-casebugs.

• BUG FIX (Archiver only): The archiver query form would sometimes generate incorrectqueries. This has been fixed.

• BUG FIX: The C storage-manager server was not 64-bit clean. It is now completely 64-bit cleanand can store huge files on 64-bit machines.

• BUG FIX: The “Wrap Lines” feature for message preview was broken on Internet Explorer inCanIt 8.0.10. It is now fixed.




• EXPERIMENTAL FEATURE: Bayes data can be retrieved from Storage Manager rather thanlocal CDB files. This feature is not yet ready for general use.

• MINOR FEATURE: When previewing a message, you can request that the browser wrap longtext lines.

• MINOR FEATURE (CanIt-Domain-PRO): API calls that retrieve realm information return thefull path from the realm to “base”.

• UPDATE: Update to ClamAV 0.97.3.

• BUG FIX: CanIt Storage Manager was not 64-bit clean and would fail to store files over 2GBon 64-bit systems. This has been fixed.

• BUG FIX: The web interface correctly validates each Verification Server if you enter more thanone for a given domain.

• BUG FIX: The Audit Trail feature now works for Global Settings.

• BUG FIX (Archiver): A query with an OR clause would not respect date restrictions properly.This has been fixed.

• BUG FIX (Archiver): The archiver would fail to parse a References: header that looked likethis: <msgid1><msgid2><msgid3>...

• BUG FIX (Log Searcher): The “what” field is a now pulldown rather than a free-form text field.

• BUG FIX (Archiver): The archive indexer could sometimes overflow an internal PostgreSQLlimit, causing indexing failures and enormous PostgreSQL logs. This has been fixed.


• NEW FEATURE: Many reports now keep statistics on message sizes as well as message counts.(For example, you can pull a report showing daily message byte traffic.) Note that the bytecount data is not available for historical data—it is only available for messages received afterupgrading to 8.0.9.

• NEW FEATURE: A general-purpose key/value storage system available via the API has beenimplemented. This can be used for integration with other applications, for storing provisioningdata, etc.

• POLICY CHANGE: Messages held for filename extension, MIME type, etc. rules are stillspam-scanned and assigned a score. This permits auto-rejection of messages that would oth-erwise be held unnecessarily. It also permits messages with Hold-policy attachments to betokenized for Bayes training.


287

• POLICY CHANGE: If a user chooses not to add voting links to whitelisted messages, this set-ting is honored for all kinds of whitelists. Previously, it was honored only for sender whitelists.

• POLICY CHANGE: “Hold” settings have been renamed “Hold/Tag” to emphasize that theynow tag the subject in a tag-only stream.

• POLICY CHANGE (CanIt-Domain-PRO only): If a realm is deleted, child realms becomechildren of the deleted realm’s parent rather than unconditionally becoming children of “base”.

• IMPROVEMENT: Many new templates have been added. These permit (for example) transla-tion of many CanIt-generated email messages.

• BUG FIX: Upper-case host names would cause cluster management to have problems. This hasbeen fixed.

• BUG FIX: Some API calls would return an empty YAML document when they really shouldhave returned a zero-element array. This has been fixed.

• BUG FIX: The templating method “T->menu items()” did not return correct values for sub-menus; this has been fixed.

• BUG FIX: The new “Rewrite” user-lookup method didn’t appear as a possible streaming methodin Domain Mappings. This has been fixed.

• BUG FIX: Some API calls mis-handled mixed-case host names; this has been fixed.

• BUG FIX: If the EHLO command fails when running a Verification Server check, we send aRSET before the HELO as per the SMTP standard.


• NEW FEATURE (Archiver only): CanIt can take a search expression and generate a zip filecontaining all archived messages that match. You can also configure CanIt to zip up archivedmail just before it expires from the archive.

• NEW FEATURE: The new “Rewrite” user-lookup permits you to map email addresses tostreams using a simple rewriting expression.

• NEW FEATURE: Old Bayes databases are removed by the nightly cron job. An “old” databaseis defined as one that has not been trained in a given timespan (by default, 365 days.)

• NEW FEATURE: A nightly system check warns if any local filesystem has less than 10% spaceremaining.

• NEW FEATURE: The rate of sending Pending Notifications can be throttled. This can helpprevent excessive CPU usage if many notifications need to be sent.

• IMPROVEMENT (Appliance only): If the Log Forwarder has trouble forwarding logs to aremote server, it logs more informative error messages to make it easier to find the problem.



• IMPROVEMENT (Domain-PRO only): Realm administrators can view both a flat list of realmsand a tree view. Previously, they could only see the tree view.

• UPGRADE: Update ClamAV from 0.97.1 to 0.97.2

• BUG FIX: The RSS Feed feature could inadvertently show incidents in other streams. This hasbeen fixed.

• BUG FIX: The “Clean Looks” theme did not include links to the manuals. This has been fixed.

• BUG FIX: The API code could occasionally mis-handle mixed-case domain names. This hasbeen fixed.

• BUG FIX: Warnings about undefined variables in the DKIM module were suppressed.


• NEW FEATURE (Archiver): The archiver permits construction of complex boolean searchqueries. You can also save queries for later re-use.


Because of the new method of query construction, the API call for searching the archive haschanged. Please read the API Guide for details.

• NEW FEATURE: You can specify whether a Verification Server entry should always queuemail if the verification server is down, or queue only for addresses seen in the last 60 days.

• PERFORMANCE IMPROVEMENT: Pending Notifications are sent with Sendmail’s SuperSafeflag set to “off”. We feel this is an acceptable tradeoff between safety and performance.

• COSMETIC FIX: The layout of the “Known Networks” page has been changed so you caneasily compare settings across networks rather than having to expand individual networks to seetheir settings.

• COSMETIC FIX: The “Viewing Stream xxx” display has been streamlined and made moreunobtrusive.

• POLICY CHANGE (CanIt-Domain-PRO only): Realm administrators are allowed to set the“Full name for sender of CanIt notifications” template.

• BUG FIX: For hosts marked “Inbound” only, we remove the normal mailertable entries thatCanIt would create based on the Domain Routing table.

• BUG FIX: The “Sharded Database” add-on component would ignore the specifieddb connect timeout. This has been fixed.

• BUG FIX: canit-failover-verify-setup.pl would sometimes incorrectly complain about a mis-configuration when in fact there was no misconfiguration. This has been fixed.


289

• BUG FIX: The API would sometimes fail if PHP’s “magic quotes gpc” setting was enabled.This has been fixed.

• BUG FIX: The Domain Routing page no longer accepts a domain of “*” (which would causerouting problems.)

• BUG FIX: The LDAP User Lookup does not insist on a non-blank Base DN (it can be emptyaccording to RFC 2253.)

• BUG FIX: If the PhishingAddress plugin action was set to “Reject” but a message scored overthe hold threshold, it would be held instead of rejected. This has been fixed.

• BUG FIX: The log-searching code could return incorrect results if you have more than one loghost. This has been fixed, but log-searching with more than one log host is slower. We willaddress the speed regression in a future release.

• BUG FIX: The failover code now copies pg xlog directories when making a base backup. Thisis required for older versions of PostgreSQL.


• NEW FEATURE (Archiver): We now include several reports such as messages and bytesarchived per day and number of email addresses with archived mail.

• NEW FEATURE (Archiver): You can ask CanIt to redeliver an archived message to the emailaddress(es) of your choice.

• NEW FEATURE (Archiver): You can now do a substring search on envelope recipients inaddition to exact matching.

• NEW FEATURE (Archiver): CanIt can now archive internal and outbound email using theSMTP Journalling feature of Microsoft Exchange 2007 and 2010. (2003 is not supported yet.)

• NEW FEATURE (Domain-PRO only): The pseudo-user *localroot* may be used to grant per-missions to realm administrators without also granting them automatically to administrators ofsub-realms.

• IMPROVEMENT: Each CanIt node periodically measures the latency to all Storage Managernodes and uses them in order of ascending latency. Thus, reads are preferably done on “close”machines, improving performance and potentially decreasing bandwidth across WAN links.(See the Administration Guide for more information and for information on overriding the orderof Storage Manager accesses.)

• IMPROVEMENT (Archiver): You can choose whether or not you wish to archived tagged mailfor tag-only streams. The default is to archive tagged mail.

• IMPROVEMENT (Archiver, Domain-PRO only): You can specify in which realms internalmachines may archive internal email.




If you have used the Authorized Hosts feature of Archiver with Domain-PRO, you need tore-enter the hosts and specify which realm(s) are applicable to each host.

• IMPROVEMENT: The log-forwarding feature now imposes timeouts on logs transported viaTCP so as not to stall log-forwarding if a log host is down.

• COSMETIC IMPROVEMENT: Large numbers in reports are displayed in a more human-readable format. In the HTML table, the human-readable format is visible if you hover overthe raw number.

• BUG FIX: All links in CanIt-generated messages have the rel=“nofollow” attribute. This is toreduce the chances of search engines crawling the links and inappropriately causing CanIt totake actions.

• BUG FIX: CanIt rejects unauthenticated actions if the user-agent is one of a number of knownsearch-engine crawlers. This prevents (for example) messages from being inappropriately ac-cepted or rejected if a notification message accidentally winds up getting indexed by Google,Yahoo!, etc.

• BUG FIX (Appliance Only): canit-system bails out immediately if it detects an upgrade inprogress. This should avoid complaints from cron during CanIt Appliance upgrades. [Unfortu-nately, the fix will only apply when upgrading from 8.0.6 to the next version.]

• BUG FIX: Some obscure edge-cases in the Web interface code that could cause redirect loopswere fixed.

• BUG FIX: The “dormant stream” report no longer reports special streams or forced-to streams.


• POLICY CHANGE: The Bayes tokenizer uses a proper HTML parser to extract tokens fromtext/html parts. This should give better results than the older naive parser.

• MAJOR NEW FEATURE: CanIt-Archiver keeps an audit trail of searches and message ac-cesses. Administrators and end-users can see the audit trail. (End-users can only see audit trailsrelating to their own archive.)

• MAJOR NEW FEATURE: CanIt-Archiver has a hook for archiving internal email that normallywould not be seen by CanIt.

• NEW FEATURE (CanIt-Domain-PRO only): Realm administrators can be granted access torealm user fields on a per-user, per-field basis.

• NEW FEATURE: For User-Lookups that cache results in the database, the default cache expiryhas been changed from 24 hours to 120 hours. A new parameter (Cache Refresh Time) causesCanIt to attempt to refresh cache entries older that 12 hours, but continue to use entries up tothe cache expiry time.


291

• MAJOR IMPROVEMENT: CanIt-Archiver compresses archived messages using “bzip2” com-pression.

• MAJOR IMPROVEMENT: CanIt-Archiver stores only one copy of duplicate messages receivedon a given day for a given realm.

• PERFORMANCE IMPROVEMENT: Bayes data is synchronized to all scanners in parallel fromthe ticker host rather than sequentially.

• UPDATE: Our appliance ISO images are now based on Debian 6.0 (“squeeze”) instead of 5.0(“lenny”)

• BUG FIX: Envelope and header senders are now compared case-insensitively to determinewhether or not to display a warning that they differ.

• BUG FIX: Messages with encoded subjects would not show up in archive searches that shouldhit words in their subjects. This has been fixed.

• BUG FIX: A rare edge-case could break statistics collection if you use Storage Manager andaccept mail over IPv6. This has been fixed.

• BUG FIX: Setting a storage manager node “read-only” works as expected; before, the StorageManager wizard did not distinguish properly between read-only and read/write nodes.

• BUG FIX: The RunBayesJournal task has been modified to ensure that it can keep up with therate of incoming training requests. If it is unable to keep up, then it trains a sample of requestsrather than all of them.


• EXPERIMENTAL NEW FEATURE: We have an email archiving component for CanIt. Thisextra-cost add-on component is available only on our Debian-based appliances and is consideredto be beta software at this point. Contact your sales representative if you are interested in testingit.

• POLICY CHANGE: CanIt now shows the header From: address rather than the envelope senderaddress in the Trap Contents display. This address is more likely to be useful for whitelisting orblacklisting than the envelope sender.

• POLICY CHANGE: If the “auto-populate notification address” setting is set to “Yes”, we do*not* set the notification address for a forced-to stream. Setting the address could result inleaking of sensitive information.

• NEW FEATURE: The “View this Stream” text box uses auto-completion to suggest streamnames.

• BUG FIX (CanIt-Domain-PRO only): You can now set a realm’s parent realm via the API.



• BUG FIX: You can now allow blacklist/whitelist sender options for “Clickable HTML” notifi-cations.

• BUG FIX: The notification template has a new %{fullurl} replacement tag that is substitutedwith the full URL of an incident.

• BUG FIX (CanIt-Domain-PRO only): A very rare edge-case could create a user that could neverbe deleted from the Web interface. This has been fixed [and all such users are deletable now.]

• BUG FIX: Timeouts for Storage Manager are now configurable and are correctly implementedfor both single-node and multiple-node installations.

• BUG FIX: The log-searching API controller would sometimes fail; this has been fixed.

• BUG FIX: Suppress use of uninitialized variable errors in Bayes tokenizer.


• NEW FEATURE (CanIt-Domain-PRO appliance only): The log-indexer can forward log lines(using the SYSLOG protocol) to remote hosts on a per-realm basis.

• IMPROVEMENT: Reduce the disk I/O consumed by the log-indexing daemon.

• POLICY CHANGE: Access to “Search Logs” is now controlled by a separate permission ratherthan the “View Trap” permissions.

• POLICY CHANGE: We have clarified the licensing terms of our data feeds such as RPTN andour RBLs. Please see the LICENSE.TXT file for details.

• POLICY CHANGE: On CanIt appliances, we now auto-create a mail alias for “root” that goesto the CanIt Administrator email address.

• UPDATE: Upgrade to ClamAV 0.97.

• BUG FIX: Package a missing cron task in canit-log-correlator.

• BUG FIX: When searching logs, correctly handle the case when the index is updated during asearch.

• BUG FIX: Fix a number of deprecation warnings with PHP 5.3.

• BUG FIX: Fix typo in theme file that caused browser to request a nonexistent CSS file.


• NEW FEATURE: In addition to rate-limiting outbound mail by sender, you can also rate-limitit by originating IP address.


293

• NEW FEATURE (Appliance only): Implemented new API calls to perform log searches. Note:The API client libraries were updated to permit query parameters on GET requests; to use thenew API calls, you will need to use the latest API client libraries.

• POLICY CHANGE: We have a new “Mixed” real-time DNS-based list. This list includes hoststhat send a large amount of both spam and non-spam. They should be penalized somewhat, butnot as much as hosts in “SpamSource”.

• GUI IMPROVEMENT: All pages display the “Pager”, “Filter” and “Enter specific object” ele-ments in that consistent order.

• BUG FIX: The audit-trail for Preferences did not work; it works correctly now.

• BUG FIX: MIMEDefang failed to install on (really old versions of) FreeBSD. This has beenfixed.

• BUG FIX: Voting links were sometimes not removed even if they should have been. This hasbeen fixed.


• BUG FIX: The log-indexer would sometimes produce incorrect timestamps when indexing logfiles. This has been fixed.

• BUG FIX: A third-party PHP module was not compatible with PHP4. We have backported it.

• BUG FIX: The new Web code broke the RSS Feed feature; this has been fixed.

• BUG FIX: A minor display problem when clicking on the message subject in a Pending Notifi-cation message has been fixed.

• BUG FIX: A very rare edge case in the PHP code that could produce illegal SQL has been fixed.

• BUG FIX: In very rare cases, a dead storage manager node could cause a scanning process toterminate unexpectedly. This has been fixed.

• BUG FIX: On appliances, clamd could be disabled unintentionally during an upgrade. This hasbeen fixed.

• BUG FIX: The Verification Server feature could sometimes misinterpret an ESMTP “SIZE”keyword from the back-end server. This has been fixed.

• BUG FIX: Remove deprecated call-time pass-by-reference instances in the PHP code.

• DOC FIX: In the Administration Guide, document the fact that the SNMP agent requires ahelper cron job to run once a minute.




• MAJOR NEW FEATURE: CanIt keeps an audit trail of all changes to settings, rules, etc. Youcan review the audit trail from most pages by clicking “Show Changes”.

• MAJOR NEW FEATURE: CanIt can perform full-text indexing and searching of all mail logs.Note that this feature is available only on our Debian appliances (“Lenny” release) and is notinstalled by default. See the Administration Guide for installation details.

• NEW FEATURE: All of the online documentation can now be searched live from the CanItinterface.

• MAJOR CHANGE: The PHP Web interface has been completely rewritten, making themingmuch easier.


IF YOU HAVE THEMED CANIT, YOU WILL NEED TO REWORK YOUR CUSTOMIZA-TIONS

• POLICY CHANGE: We no longer provide binary RPMs for SuSE Linux Enterprise Server 9.

• POLICY CHANGE: Default database connection timeout has been increased from 10 secondsto 20 seconds. 10 was causing problems on some systems.

• POLICY CHANGE: In a Verification Server entry, if you choose “Queue” rather than “Temp-fail” when the back-end server is down, CanIt only permits recipients who have been seen withinthe last 60 days. It tempfails any others. This makes it much safer to use “Queue” with muchless risk of backscatter.

• UPDATE: Update ClamAV from 0.96.4 to 0.96.5.

• NEW FEATURE: The Pending Notification page has a button to send a notification immedi-ately.

• NEW FEATURE: We now have IPv6 geolocation data (though it is less granular than the IPv4data, listing only the country.)

• NEW FEATURE: When the failover code fails over, it executes all scripts in/usr/share/canit/failover/notify.d/ This lets you write scripts to send administrators notice thatthe database has failed over.

• NEW API CALLS:

/realm/@@/streams/with pending: List streams with new pending incidents.

/realm/@@/stream/@@/pending flag: Get/set the Pending Notification flag.

/realm/somerealm/realm mappings: Get all realm mappings for a given realm.

• API IMPROVEMENT: The API returns more information about incidents than before, includ-ing sender, recipient, host, etc.


295

• IMPROVEMENT: (CanIt-Domain-PRO only): A configuration file setting lets you use AJAX-enabled auto-completing text fields wherever a realm entry box appears. The AJAX code hasbeen optimized since the previous release and should be much faster, especially for the siteadministrator.

• IMPROVEMENT: Verification Server SMTP callbacks understand the ESMTP “SIZE” key-word.

• IMPROVEMENT: You can control caching of invalid recipients and invalid recipients usingmemcached independently. (For example, you can cache valid recipients, but not invalid ones.)

• IMPROVEMENT: The Permissions page forces you to enter a stream class if there isn’t oneinstead of silently ignoring input.

• IMPROVEMENT: The POP3 authentication method would fail against Exchange 2007 andnewer because of a Microsoft bug. We have code in CanIt to work around Microsoft’s bug.

• COSMETIC IMPROVEMENT: The charts displayed in Reports have been improved: Redun-dant trailing zeros are deleted and all load charts are lined up.

• COSMETIC IMPROVEMENT: The User Lookup test page has been made clearer. Tests thatcan’t work for a particular user lookup method are suppressed.

• BUG FIX: If the system has been configured to force user-names to lower-case, then all com-ponents of the system enforce that setting on data entry.

• BUG FIX: When CanIt stripped out existing training links from messages, it would sometimesbe a bit greedy and strip out too much of the message. This has been fixed.

• BUG FIX: SNMP monitoring code has been rewritten and cleaned up. Permissions problemswith monitoring PostgreSQL have been fixed.


The SNMP code has changed to require only a single SNMP agent process. If you have config-ured SNMP, you will need to reconfigure it to use the new SNMP agent.

• BUG FIX: sendmail-account-info.pl could occasionally fail. (This affects very few people...)

• BUG FIX: Update the Python API client library to work with Python 2.6 and later.

• BUG FIX: In CanIt-Domain-PRO, the Domain Setup Wizard would sometimes create a Verifi-cation Server entry in the base realm instead of the appropriate realm. This has been fixed.


• NEW FEATURE: CanIt now has built-in support for DKIM (DomainKeys Identified Mail). Seehttp://dkim.org for a description of DKIM.


http://dkim.org


• NEW FEATURE: We have experimental support for Vouch By Reference (RFC 5518). We maychange the way it is implemented in a future release of CanIt, but will provide an automaticupgrade path.

• IMPROVEMENT: If you are using outbound rate limiting *and* use SMTP AUTH on the CanItserver, we limit outbound mail based on the authentication name as well as the purported enve-lope sender. This is to avoid spammers bypassing rate limiting by changing the sender address.

• NEW API CALLS: GET /realm/xx/stream/yy/addresses seen and GET /realm/xx/domains seenreturn statistics about observed email addresses per stream/realm.

• IMPROVEMENT: Graphical reports now show statistics for “bad” things (eg spam and viruses)using a reddish palette and “good” things using a greenish palette.

• IMPROVEMENT: The failover scripts include additional checks to prevent certain misconfigu-rations.

• IMPROVEMENT: You can specify a database connection timeout. This improves response timeif you have configured a databaseless-filter mode and the database is down. The default timeoutis 10 seconds.

• UPGRADE: Updated ClamAV to version 0.96.4.

• PLATFORM SUPPORT: We have dropped support for RPMs for Red Hat Enterprise Linux 3.We have dropped binary package support for Debian Sarge (3.1).

• POLICY CHANGE: The CanIt Appliance setup screen will not let you name your CanIt host“something.local” or “something.localdomain”.

• BUG FIX: The username “defang” was hard-coded in a few places. Now everything respectsthe setting in canit.conf.

• BUG FIX: Verification Servers were documented as taking a “/port” suffix. Now that actuallyworks!

• BUG FIX: In the API server, using ’@@’ for realm and/or stream now works as documented.

• BUG FIX: A long-standing bug that could cause Pending Notifications to fail if there are pend-ing messages with invalidly-encoded subjects has been fixed.

• BUG FIX: The old message-hashing algorithm could sometimes consider two different mes-sages to be the same incident. This has been fixed, but there will be a transition period ofseveral days for your CanIt installation to move completely to the new algorithm. (We mustretain the old algorithm until all incidents hashed with it have expired.)

• MISCELLANEOUS: Many minor bug fixes and code cleanups.


297


• POLICY CHANGE: Add default rules for the Roaring Penguin reputation lists.

*** NOTE *** If you are already using our lists, please check the RBL rules in the defaultstream (base realm in Domain-PRO) carefully after upgrading!

• NEW FEATURE: You can configure CanIt to create an incident for all messages, even non-spam ones. The feature is disabled by default and we do not recommend enabling it on busysites.

• NEW FEATURE / POLICY CHANGE: The default notification format has been changed to“HTML with Links” instead of “Clickable Webform”. The “Clickable Webform” form fails inmany email clients.

• NEW FEATURE: Add a button to the Preferences : Notification page that allows a user torequest an immediate notification.

• UPDATE: Prepare CanIt to work properly with forthcoming PostgreSQL 9.0.0.

• MINOR IMPROVEMENT: Add a “tag” field in Master RBL List. This tag (if present) is usedin log messages rather than the long RBL name.

• IMPROVEMENT: Add more convenient do get, do put, do delete and do post methods to PerlAPI client library.

• IMPROVEMENT (CanIt-Domain-PRO only): Instead of a pull-down list of realms, youcan use an AJAXy auto-completion text field in the “Switch Realm” box. See $Con-fig[’RealmSelectWidget’] in the config-domain-pro.php file.

• BUG FIX: The API server includes the “parent” field of each realm in the “GET /realms” result.

• BUG FIX: Previously, the API server did not convert a user name of ’@@’ to the logged-inusername. Now it works as advertised.

• BUG FIX: The Known Networks page would sometimes display a spurious error message ”Youcannot use Force To Stream on a network containing 127.0.0.1 or ::1” This has been fixed.

• BUG FIX: Uploading an SSL certificate under Setup : HTTPS would put the certificate on allcluster members. Now it only puts it on the particular web server that your browser communi-cates with.

• BUG FIX: When disposing of locally-held messages, correctly report the sending relay to theRoaring Penguin Reputation Collection system.

• BUG FIX: Properly implement the retry-delay for dead Storage Manager nodes.

• BUG FIX: Use the –compress flag on all “rsync” commands.

• BUG FIX: Do not check RPTN data for freshness on cluster members that are not marked as“Sync Bayes?” nodes.




• NEW FEATURE: CanIt can stream inbound messages by directly injecting files into the Send-mail client queue. This can significantly improve performance and reduce disk I/O when stream-ing messages.

• IMPROVEMENT (CanIt-Domain-PRO only): Realm administrators now have access to theGreylisting Report.

• POLICY CHANGE: SORBS has been removed from the suggested set of useful RBLs.

• BUG FIX: The “Known Networks” cache size has been increased to avoid “ping-ponging” thecache in certain situations.

• BUG FIX: The sample PHP API client code now implements DELETE.

• BUG FIX: The CanIt-Connectwise integration module no longer uses the DateTime module.

• BUG FIX: Subject lines in the Pending Notification messages would sometimes be convertedto UTF-8 incorrectly. This has been fixed.


• BUG FIX (CanIt-PRO only): Brand new installations would fail with a PHP error. This hasbeen fixed.


• BUG FIX: On CanIt-PRO only, if the login/password were the default “admin/canit”, the systemwould fail with a fatal PHP error. This has been fixed.


• UPDATE: Update MIMEDefang to 2.70

• NEW FEATURE: Scanners can be marked as “Inbound” and/or “Outbound”. You can also marka scanner as not needing Bayes data if it’s only used for outbound scanning.

• IMPROVEMENT: You can specify whether an RBL applies to IPv4 addresses, IPv6 addressesor both.

• IMPROVEMENT: The Bayes calculation handles edge-cases better rather than biasing themtowards “spam”.

• IMPROVEMENT: The stream (and possibly realm) are included in the URLs generated byPending Notifications.


299

• CHANGE: The “User”, “PID File” and “Root Directory” settings for Storage Manager are nowspecified in canit.conf rather than being stored in the database and updated via the Web interface.This allows you to use different values on different machines, and also makes the canit-systemstartup script more robust in the face of a missing database.

• CHANGE: “Sender-Whitelisted” messages are reported to RPTN (but not trained locally.)

• CHANGE: We no longer track the “Expired from Trap” statistic. Instead, when an incident iscreated, it increments a new “Quarantined” statistic.

• BUG FIX: IPv6 addresses were not always correctly parsed out of Received: headers; this hasbeen fixed.

• BUG FIX: Avoid useless DNS lookups in CanIt::Socket.

• BUG FIX: Make notifications stream settings accessible via API.

• BUG FIX: Auto-whitelisting could inadvertently create mixed-case sender rules. This has beenfixed.

• BUG FIX (CanIt-Domain-PRO only): A realm’s “default” stream always inherited frombase:default even if the admin had explicitly turned off inheritance. This has been fixed.

• BUG FIX: The LDAP lookup Perl code broke with very new versions of Net::LDAP. This hasbeen fixed.

• BUG FIX: The test for Blacklisted Recipients was case-sensitive; this has been fixed.

• BUG FIX: If you had a wildcard Verification Server, it would sometimes be used even if therewas a more-specific entry. This has been fixed.

• BUG FIX: Country Code rules can now be exported to CSV files and imported from CSV files.

• BUG FIX: The Web interface formerly took quadratic time to obtain the tree of realms; it nowtakes linear time.

• BUG FIX: Provide API-level access for setting a stream’s parent.

• BUG FIX: Improve CanIt API server handling of JSON vs YAML.


• IMPROVEMENT: HTTPS (with self-signed certificates) is enabled on appliances by default.

• DOCUMENTATION FIX: Fix typo in Administration Guide Memcached configuration instruc-tions.

• BUG FIX: Allow a wildcard SPF rule (broken in 7.0.0).

• BUG FIX: Add workaround for ancient Perl LWP library shipped with Red Hat EnterpriseLinux 4.



• BUG FIX: Periodic Reports were broken on CanIt-PRO. They have been fixed. Note: If you areupgrading from pre-7.0.0, you may still have to edit all of your charts and save them to updatethe stored report configuration.

• BUG FIX: Avoid warning caused by ancient version of PostgreSQL shipped with Red HatEnterprise Linux 4.

• BUG FIX: Only allow selection of LDAP (Active Directory) when first creating a User Lookup.Thereafter, it becomes (and stays) LDAP (Generic).

• BUG FIX: Suppress pointless warning when entering a Verification Server of “ignore”.

• BUG FIX: Prevent startup code from always registering a new cluster member as a standalonemachine.

• BUG FIX: Don’t attempt DNS lookups on hostnames that are already IPv4 or IPv6 addresses.

• BUG FIX: 9-digit old-style incident IDs would cause the work journal task to fail. This hasbeen fixed.

• BUG FIX (CanIt-Domain-PRO only): Allow Base URL of CanIt Installation to be set on aper-realm basis.


• UPDATE: Update to ClamAV version 0.96.

• BUG FIX: Remove the long-obsolete “Sendmail” domain-mapping option.

• BUG FIX: Subject lines with NUL characters could produce badly-rendered Pending Notifica-tion reports. This has been fixed.

• BUG FIX: Fix compilation error in Storage Manager on Gentoo.

• BUG FIX: Avoid spurious warning in database upgrade script.

• BUG FIX: Include correct text in incident report when sender is whitelisted due to SMTPAUTH.

• BUG FIX: Fix the optional “Next Msg” and “Prev Msg” links in the incident details page; thesewere broken by the 7.0.0 release.

• BUG FIX: In 7.0.0, a user-lookup method whose name matched the method name would fail.This has been fixed.

• BUG FIX (CanIt-Domain-PRO only): The 7.0.0 upgrade accidentally reduced the permissionsof realm administrators. This has been fixed.

• BUG FIX: The Known Networks page produced invalid HTML; this has been fixed.

• BUG FIX: The database upgrade script could fail on certain databases that were upgraded fromold versions of PostgreSQL. This has been fixed.


301


• MAJOR NEW FEATURE: CanIt can automatically block senders who send too many messagesper hour. This rate-limiting is controlled by a Known Networks flag and can be used to detectand block internally-compromised email accounts.

• MAJOR NEW FEATURE (CanIt-Domain-PRO only) Realms can now be hierarchical. Thisallows many levels of administrative control; a customer in charge of a realm can be allowed tomanage sub-realms.

• MAJOR NEW FEATURE: CanIt installations collect data about IP address reputation and sendthe data back to Roaring Penguin Software. This will be used to build a set of DNS-basedblocklists usable by CanIt customers. NOTE: You should open UDP port 6568 outbound so theCanIt machines can report the IP reputation data.

• MAJOR CHANGE: The API server has been rewritten in PHP. As a result, it is much easierto deploy and does not need FastCGI or Catalyst. Also, if you choose, you can make the APIavailable to realm administrators or even end-users. (Normal permission checks apply.)


The API version number has changed from 1.0 to 2.0. You should rework any scripts that usethe API and make sure they still work correctly.

• MAJOR CHANGE: Incident IDs are no longer integers, but string identifiers. This is to sup-port a future add-on component that allows CanIt incident data to be spread across multiplePostgreSQL servers.

• MAJOR NEW FEATURE: Roaring Penguin Software Inc. provides four new DNSBLs to CanItcustomers; see the Administration Guide for details.

• NEW FEATURE: Master RBL’s can be marked “Block” or “Allow”, which controls the RBLrules that can be created.

• NEW FEATURE: You can extend the enforced greylisting “quiet time” for hosts listed on aDNSBL.

• NEW FEATURE: (CanIt-Domain-PRO only) You can store up to four user-defined pieces ofinformation per realm.

• ENHANCEMENT: DNSBLs now let you specify an A record to match or a bitmask to maskagainst. This lets CanIt handle combined DNSBLs that return multiple pieces of informationencoded in the A record.

• UPGRADE: Upgraded SpamAssassin from 3.2.5 to 3.3.0.

• IMPROVEMENT: You can set a timeout on Verification Server lookups and User Lookups.

• IMPROVEMENT: “Subject” custom rules apply to both raw and decoded subject lines.



• IMPROVEMENT: CanIt Storage Manager packs old data into CDB databases; this reduces thenumber of files in the Storage Manager Tree making it easier to back up and consuming fewerinodes.

• IMPROVEMENT: The Sanity Checker module checks for many more problems and misconfig-urations.

• IMPROVEMENT: The Dictionary Attack Detector uses a Known Networks flag to avoid ban-ning friendly hosts. This replaces the older text entry box with a list of hosts.

• IMPROVEMENT: The “Clickable Webform” Pending Notification has been improved so largepending lists don’t generate over-long URLs. Also, all subject lines are decoded and presentedin UTF-8.

• IMPROVEMENT: The failover code refuses to fail over if the standby database is active forsome reason.

• IMPROVEMENT: WAL-file copying in the failover code has been made more robust.

• IMPROVEMENT: System Load graphs are now in a zoomable vector format on browses thatsupport the HTML Canvas tag (this means any modern browser except Internet Explorer.)

• IMPROVEMENT: The API always returns a stream’s parent when returning stream data.

• IMPROVEMENT: DNSBL descriptions are shown in the Spam Analysis Report.

• IMPROVEMENT: We don’t use DB File unless we actually encounter a Berkeley DB file. Thiscan reduce memory usage.

• IMPROVEMENT: French and Portuguese translations have been overhauled.

• SECURITY ENHANCEMENT: The “goto” redirection parameter is sanitized to avoid cross-site scripting attacks.

• IMPROVEMENT: (Appliance only) The curses-based “setup” utility lets you reset CanIt userpasswords.

• IMPROVEMENT: (CanIt-Domain-PRO only) Many formerly-global settings like the CanIt ad-ministrator email address are settable on a per-realm basis.

• PERFORMANCE IMPROVEMENT: You can use memcached to cache Verification Server re-sults.

• MINOR NEW FEATURE: You can purge all rules and settings from a stream.

• MINOR NEW FEATURE: You can remove permission for end-users to disable stream inheri-tance.

• MINOR IMPROVEMENT: Text in the “Strip Attachments” notification is now templatable.

• MINOR IMPROVEMENT: You can set the default action for the “Clickable Webform” notifi-cation, and also add “Blacklist/Whitelist Sender” options to the notification.


303

• MINOR IMPROVEMENT: You can tell the LDAP lookup not to validate the server certificate(if, for example, it uses a self-signed certificate.)

• MINOR IMPROVEMENT: System Check test names are hyper-linked to descriptions in theAdministration Guide.

• PERFORMANCE IMPROVEMENT: The Storage Manager server uses “sendfile” to send dataif possible. Otherwise, it tries “mmap” and only as a last resort falls back to “read/write”.

• CLEANUP: The database schema has been cleaned up to improve performance and maintain-ability.

• GUI CLEANUP: The Pending Trap displays incident dates in a more readable way.

• GUI CLEANUP: The Known Networks interface has been reworked to avoid very wide pagesthat require side-scrolling.

• POLICY CHANGE: The “Handling for Windows Executables” setting has been removed. In-stead, use Filename Extension rules. On upgrade, appropriate Filename Extension rules arecreated to keep the same behaviour as the pre-upgrade version.

• POLICY CHANGE: The “Secondary MX Machines” setting has been removed. Instead, useKnown Networks flags. On upgrade, appropriate Known Networks entries are created.

• POLICY CHANGE: We no longer tokenize Microsoft Word documents. They were leading totoo many false positives. This change may be revisited in a future release.

• POLICY CHANGE: The “Database Cron Runner” flag in Cluster Management has been ig-nored since release 6.1.0. The flag has therefore been removed.

• BUG FIX: We don’t count addresses in addresses seen unless a domain is known to validaterecipients.

• BUG FIX (Appliance only): The “Set Timezone” menu option now works properly and actuallysets the time zone.

• BUG FIX: Bayes training is more robust in the face of corrupt CDB files.

• BUG FIX: Known Networks would refuse to allow a “Force-to-Stream” value for SMTP-AUTH. This has been fixed.

• BUG FIX: The “Top N” reports in long-term statistics used to issue PHP errors on PHP 5.3; thishas been fixed.

• BUG FIX: CanIt would incorrectly force local parts of email addresses to lower-case when do-ing verification server lookups. This can break things like SRS, so we leave the local part alonenow. (For the purposes of rules, however, the local part is still compared case-insensitively.)

• BUG FIX (Domain-PRO only): Realm Administrators are now given full access to traps withintheir realms.

• BUG FIX: Daily reports were broken on PHP 5.3.



• BUG FIX: Remove all PHP calls to the deprecated “ereg*” functions in favour of “preg*”

• BUG FIX: (CanIt-Domain-PRO appliance only) Deleting a realm also deletes domain routesassociated with the realm.

• BUG FIX: SPF “permerror” and “temperror” returns codes are handled properly.

• BUG FIX: Oversize text/plain parts are no longer scanned with SpamAssassin.

• BUG FIX: The system is much more robust in the face of corrupt Bayes CDB files.

• BUG FIX: canit-failover-verify-setup.pl would report a spurious test failure.

• BUG FIX: CSS stylesheets have been fixed up to have more consistent font selection.


• BUG FIX: Make startup code regenerate mailertable and access databases on appliances.

• BUG FIX: Make startup code coexist more peacefully with PgBouncer.

• BUG FIX: Fix errors in IPv6 validation in Rules : Networks.

• BUG FIX: Force scanners to notice changes to Known Networks immediately.

• BUG FIX: Make “PUT /domain routes/activate” API command actually work.

• MINOR BUG FIX: Suppress warnings that cron job has not run on new installations; we onlytrigger the test after system has been installed for at least a day.

• MINOR BUG FIX: Fix possible print formatting error in Custom Rule test.

• MINOR BUG FIX: Suppress “Use of uninitialized variable” warning in Bayes code.

• MINOR BUG FIX: Make Bayes sync work on symbolically-linked source directories.


• BUG FIX: Total token counts for local streams were being reset to zero. This has been fixed.Note that no Bayes training was lost; only the token counts in the PostgreSQL database wereaffected. As streams undergo Bayes training, the token counts will be corrected automatically.


• BUG FIX: On certain systems, the database upgrade code would fail. This has been fixed.

• BUG FIX: In some cases, depending on how it was sorted, the trap display would produce anSQL error. This has been fixed.

• BUG FIX: Storage manager refused to compile on NetBSD; this has been fixed.


305


• NEW REPORT: A “Dormant Streams” report lists all streams that have not received mail in thelast 60 days.

• NEW FEATURE: “Host” rules have been replaced by “Network” rules, which can apply toCIDR blocks as well as individual hosts.

• NEW FEATURE: We have *experimental* support for IPv6. Anywhere an IPv4 address can beused, so can an IPv6. And anywhere an IPv4 CIDR can be used, so can an IPv6 CIDR. Pleasenote that there may be many untested edge cases, hence the designation “experimental”.

• NEW FEATURE (CanIt-Domain-PRO only): Realms can have an expiry date; when it nears,realm administrators get a warning. This lets hosting providers keep track of when customerservices are to expire.

• MINOR NEW FEATURE: The trap display can be sorted by the domain of the sender.

• DOCUMENTATION FIX: The theming guide has been overhauled. It’s now linked from Setup: Wizards (just like all the other manuals.)

• CHANGE: The “Incident Note” feature has been removed. It cluttered the interface and isalmost useless in CanIt-PRO and Domain-PRO.

• CHANGE: Internally, we use Mail::SPF rather than the deprecated Mail::SPF::Query to handleSPF lookups.

• POLICY CHANGE: When looking for SPF scoring rules, we recursively strip off domain com-ponents in the same way as for Domain Action rules.

• IMPROVEMENT: There is no need to manually create and maintain a script to synchronizeBayes data files. Instead, the CanIt cluster system automatically synchronizes Bayes data to allscanners.

• IMPROVEMENT: The text of the “Periodic Report” e-mail can be templated.

• IMPROVEMENT: We tokenize the “HELO” string for Bayes.

• MAJOR INTERNAL CHANGE: The internal mechanism used to run tasks across cluster mem-bers has been drastically overhauled and should be much more robust.

• PERFORMANCE IMPROVEMENT: CanIt can now use the PgBouncer connection pooler toreduce load on the database. PgBouncer is packaged for our Debian-based appliances.

• PERFORMANCE IMPROVEMENT: We use the “CDB” database format for storing Bayesdata rather than Berkeley DB. CDB should be faster than Berkeley DB and the files are portableacross operating systems and CPU architectures.

• IMPROVEMENT: If a message is oversize, we attempt to scan it anyway after removing non-text parts. (If the remaining text parts are still oversize, we do not scan the message for spam.)This should help considerably in catching spams that are artificially inflated with image attach-ments.



• BUG FIX: CanIt is now compatible with PostgreSQL 8.4.0.

• BUG FIX: The “Bogus MX” check now considers 0.0.0.0 to be bogus.

• BUG FIX: A performance regression when viewing very large traps has been fixed.

• BUG FIX: Crash-inducing typos in the SPF Rules page and the daily-mail-by- realm report havebeen fixed.


• NEW FEATURE: Sessions can be made to last longer than 8 hours with the new “RememberMe” checkbox on login screen. (Default Remember Me time is one week.)

• NEW FEATURE: The POP3, IMAP and Program external authentication methods let you stripthe domain name from the login name to generate the home stream. For example, you canconfigure it so that “[email protected]” is placed in a home stream called “user”.

• NEW FEATURE: An administrator (or realm administrator) can choose to allow senderwhitelisting/blacklisting directly from notification messages. The administrator can also setthe default pulldown settings in notification messages to “Reject” or “Do nothing”.

• NEW REPORT: An administrator (or realm administrator) can pull a report showing the numberof addresses seen per stream.

• IMPROVEMENT: In the “Clickable Webform” notification, clicking on the message subjectdisplays the message body without requiring logging in.

• CHANGE: canit-storage-manager has an official IANA port number (6568). The default porthas been changed to reflect that.

• FIX: The 70 sare stocks.cf ruleset is obsolete and has been deleted.

• BUG FIX: The API server would sometimes return a 500 error code instead of a 404 not foundcode.

• BUG FIX: If an incoming message has an X-Spam-Flag: header, we delete it.

• BUG FIX: /etc/init.d/canit-system behaves more reliably if the database happens to be downwhen it is run.

• BUG FIX: When we download a new ruleset, we now signal all scanners to re-read the rulesfiles.

• BUG FIX: Several other minor bugfixes and cosmetic improvements.


307


• UPDATES: Updated to MIMEDefang 2.68 and ClamAV 0.95.1.

• PERFORMANCE IMPROVEMENT: When using Embedded Perl, scanner startup time is im-proved. Also, more memory can be shared among scanners, reducing the total memory foot-print.

• IMPROVEMENT: In CanIt 6.0.0 and 6.0.1, incidents that expired out of the trap were nevercounted in daily statistics. Now they are counted in their own category (“Expired from Trap”)

• PERFORMANCE IMPROVEMENT: Performance of greylisting was improved on large andbusy clusters by partitioning the greylisting table in the database.

• POLICY CHANGE: If the score for AutoRejectNoIncident is lower than AutoReject, we in-crease it to match AutoReject.

• IMPROVEMENT: canit-prepare-system warns if it notices that SELinux is enabled. It also setsreasonable defaults for mx maximum in canit.conf.

• NEW SETTINGS: canit.conf has new settings in the [mimedefang] section: con-serve descriptors, md required fds and mx required fds. See /usr/share/canit/canit.conf for de-tails.

• BUG FIX: Several typos in the HTML manuals were fixed.

• BUG FIX: The address-count-by-domain report used SQL that didn’t work on old versions ofPostgreSQL; this has been fixed.

• BUG FIX: The Domain Setup Wizard would refuse to let you choose “Other” for the streamingmethod; this is fixed.

• BUG FIX: A minor rendering error on the Bayes Rules page was fixed.

• BUG FIX: The “Bulk Entry” page performs basic validation of entered data.

• BUG FIX: A bug in the Storage Manager Wizard that would only let you change one host’sStorage Manager Settings has been fixed.

• BUG FIX: PDF pie charts would render incorrectly if there were many entries; this has been(partially) fixed so that even if some entries are truncated, the pie graph displays correctly.

• BUG FIX: The “PhishingAddress” test worked, but put a nonsensical value in the list of firedtest names. This has been fixed.


• NEW FEATURE: CanIt can produce reports showing the number of valid e-mail addresses seenper domain (and per-realm, for CanIt-Domain-PRO.)



• BUG FIX: In certain very unusual situations, the database upgrade script could abort. This hasbeen fixed.

• BUG FIX: The Storage Manager wizard in 6.0.0 did not work correctly with multiple storagemanager nodes; this has been fixed.

• BUG FIX: If you create a periodic report with no charts, CanIt used to produce invalid PDF.Now, it produces a single-page PDF containing an error message.

• BUG FIX: On new installations only, the RunBayesJournal background task would die. Thishas been fixed.

• BUG FIX: A typo in version 6.0.0 would sometimes cause a filtering process to terminate ab-normally. This has been fixed.

• BUG FIX (CanIt-Domain-PRO only): Deleting a realm would sometimes leave some realmdata in the database. Now, it is all cleaned out.

• BUG FIX: Reports now have fields for realms and streams, rather than Yes/No fields “Show AllRealms” and “Only This Stream”.

• BUG FIX: Non-root users could not create periodic reports; this has been fixed.

• BUG FIX: Under VMWare, the master multiplexor process could consume a lot of CPU time.This has been fixed (but we still do not recommend running CanIt under VMWare, especiallythe PostgreSQL database server.)


• UPGRADE NOTE: It is no longer possible to upgrade versions of CanIt less than 4.0.0 to thecurrent version. If you are running CanIt older than 4.0.0, you must first upgrade to 5.0.2 beforeupgrading to 6.0.0.

• NEW FEATURE: CanIt blocks mail to or from addresses on a dynamically-maintained phishingaddress list. This list is distributed several times a day over the RPTN distribution channel.

• NEW FEATURE: On an emergency, per-domain basis, you can block Delivery Status Notifica-tions to cope with severe backscatter. (This feature is dangerous, so must be explicitly enabledunder Setup : Features)

• NEW FEATURE: CanIt can generate and e-mail PDF reports on a periodic basis. You canconfigure which reports you want and how often you want to receive them.

• POLICY CHANGE (Appliance Only): If your sources.list file contains a non-RoaringPenguinrepository, automatic updates are suppressed; you have to run the update manually in this case.

• POLICY CHANGE: The nightly RPTN download submits some statistics back to the RPTNserver for Roaring Penguin’s monitoring and analysis purposes. In particular, it submits thePostgreSQL version, license key, operating system name and version, CanIt version, count of


309

number of hosts in your cluster, count of number of valid inbound email addresses and domainsseen in the last 60 days, and daily cluster load statistics. No personally-identifying informationis reported back.

• PERFORMANCE IMPROVEMENT: If you are using Storage Manager, we do not store any-thing in PostgreSQL for a Bayes signature unless it is trained. On busy systems, this can con-siderably reduce the load on the database.

• BUG FIX: Fixed some rendering errors in the Web interface on Opera and Chrome.

• BUG FIX: RPTN downloads now validate the server certificate against Roaring Penguin’s cer-tification authority file.

• BUG FIX: The Storage Manager Wizard is better integrated with the Cluster Management GUI.

• BUG FIX: canitd, the CanIt Daemon, has been completely rewritten to improve reliability.

• BUG FIX: You can now specify a port number if PostgreSQL is listening on a non-standardport.

• BUG FIX: Silenced annoying (but harmless) log messages about duplicate keys from theLogLoad daemon task.


• MAJOR POLICY CHANGE: Self-whitelists are ignored. That is, if the sender e-mail addressis the same as the recipient e-mail address, any whitelist for that address is ignored. Similarly,if the sender domain is the same as the recipient domain (or a subdomain thereof), then anydomain whitelist is ignored.

• NEW RULES: We ship a SpamAssassin plugin that detects many kinds of targeted phishingattempts (known as “spear phishing”). Look for the RP PHISH rule in incident reports.

• PERFORMANCE IMPROVEMENT: Performance of Bayes training SQL queries has been im-proved.

• IMPROVEMENT: We do not auto-whitelist messages if the outgoing message looks like anout-of-office auto-reply.

• BUG FIX: “Import Rules” did not correctly import rules exported by “Export Rules”; this hasbeen fixed.

• BUG FIX (Domain-PRO only): Only the site administrator can switch into the special “@@”streams (which are always in the base realm.)

• BUG FIX: The MIMEDefang SNMP agent reported inaccurate data; this has been fixed.

• BUG FIX: If the master “canitd” daemon lost connection with the PostgreSQL database, itwould incorrectly stop various daemon tasks. This has been fixed.



• BUG FIX: Under certain circumstances, CanIt would inappropriately store an additional copyof released messages. This has been fixed.

• BUG FIX: (Appliance Only) Automatic upgrades were broken in 5.0.1; they are fixed in 5.0.2.(Note that we released interim 5.0.1 packages that fixed the problem also, so very few appliancesshould be affected.)

• BUG FIX: The Sendmail address mapper Path in the database was not updated to reflect thenew location of sendmail-account-info.pl. It is now.


• NEW RPMS: We now supply RPMs for Fedora 10 on i386. NOTE: This will be the LASTversion of Fedora for which we will supply RPMs. For future Fedora releases, you will need toinstall CanIt from source.

• UPDATE: ClamAV updated from 0.94.1 to 0.94.2.

• IMPROVEMENT (Appliance Only): The upgrade process on appliances has been improved;e-mail notifications are more meaningful. You can configure Automatic vs Manual upgrade viathe Web interface. The system will refuse to do automatic upgrades on a cluster.

• COSMETIC IMPROVEMENT: If the geolocation code cannot determine the location of a relay,a special small flag is shown rather than “Location Unknown”.

• BUG FIX: The RPMS for Red Hat Enterprise Linux 3 did not work because of the ancientversion of Perl on RHEL3. We have since made the code work.

• BUG FIX: The “Reset Inheritance” button for stream settings did not work in 5.0.0. It nowworks as designed.

• BUG FIX: We inadvertently used a PHP function only available in newer versions of PHP. Thisbroke some reports. We’ve fixed the code to use functions available in all supported versions ofPHP.

• BUG FIX: The sanity-checker emitted false reports of failed ticker tasks. This has been fixed.

• BUG FIX: New cluster members automatically register themselves as scanners. While this maynot always be the case, it almost always is and is a better default behaviour.

• BUG FIX: The country-name selection menu on the Rules : Countries page was too narrow inInternet Explorer. This has been fixed.

• BUG FIX: The CanIt API server would ignore realm restrictions for certain stream-listingqueries. This has been fixed.

• BUG FIX: Advanced Search would sometimes double-escape input data, leading to failedsearches that really should have succeeded.

• BUG FIX: Various minor PHP and JavaScript warnings have been fixed.


311


• MAJOR NEW FEATURE: CanIt determines the country in which a sending relay is located.You can make rules based on sending country; geolocation information is also used as Bayestokens.

• MAJOR NEW FEATURE: Cluster management has been completely revamped. It is much easyto set up a cluster now.

• MAJOR NEW FEATURE: The cluster-management system collects performance data for allscanners in the cluster; the Web interface lets you plot the data minute-by-minute, hour-by-houror day-by-day.

• MAJOR NEW FEATURE: CanIt can do dictionary-attack detection and block abusive hosts atthe firewall level. This feature is ONLY available on Linux.

• MAJOR NEW FEATURE: The Web interface includes a Domain Setup Wizard that walks youthrough all the major steps required to set up a new domain.

• NEW FEATURE: You can set expiry dates on most rules. This lets you avoid “rule creep” asmany rules accumulate and last forever.

• NEW FEATURE: The Administration Guide and Users Guide are available in HTML format.Most CanIt GUI pages link to corresponding manual sections.

• NEW FEATURE: Auto-whitelists now expire (by default after 180 days).

• NEW FEATURE: SNMP tools are packaged with CanIt. Note that you need to install andconfigure net-snmp yourself to enable the SNMP tools.

• IMPROVEMENTS: Many additional reports were added; existing reports were made more con-figurable.

• MAJOR IMPROVEMENT (Appliance Only): The text-based interface for setting up a CanItappliance has been completely rewritten and is much more usable and stable.

• MAJOR IMPROVEMENT: Sender and Domain rules work both with the SMTP envelopesender and the address in the From: header. The previous behaviour of ignoring From: wasvery confusing to end-users.

• MAJOR IMPROVEMENT: The “sanity checker” that checks for common misconfigurationshas been revamped. It now e-mails the CanIt administrator if it discovers problems.

• MAJOR CHANGE: The “ticker” is gone. Replacing it is the CanIt daemon “canitd” that runson all hosts in a cluster.

• MAJOR IMPROVEMENT: The different CanIt startup scripts have been unified into/etc/init.d/canit-system which starts or stops all processes required on a particular node.

• IMPROVEMENT: You can set “Allow Unauthorized Voting” on a per-stream (hence per-realmin Domain-PRO) basis.



• POLICY CHANGE: SURBL is now charging large users for access to SURBL data. Pleaseread the comments in /etc/mail/spamassassin/72 score compensate.cf. Similar comments applyto the URIBL.COM URI blocklist.

• POLICY CHANGE: Support terms have been changed. We have increased our excess supportfee from $75/hour to $100/hour, and have added the following clause:

After Hours Support: If mail delivery is interrupted, we reserve the right to make minimalchanges to get mail flowing again (including disabling filtering entirely) until our normal officehours, at which time we will attempt to make a complete correction.

• POLICY CHANGE: By popular demand, the “Whitelisted Action” for filenames and extensionsalso applies if a domain or host is whitelisted, and not only if an actual sender is whitelisted.

• POLICY CHANGE: We avoid greylisting very large messages (to conserve bandwidth)

• UPDATE: Updated ClamAV to version 0.94.1.

• CLEANUP: SpamAssassin rules shipped with CanIt were cleaned up and updated.

• CHANGE: The “Sendmail” address-mapping method has been removed. The upgrade processreplaces any Sendmail methods with an equivalent Program method.

• CLEANUP: Many global variables that had identical per-stream variables have been removed(the globals were really only necessary for plain-CanIt and are not needed for CanIt-PRO orCanIt-Domain-PRO.)

• CLEANUP: Several global variables that were of marginal use have been removed and therecommended behaviour has been hard-coded.

• CLEANUP: Many scripts and paths have been moved. We place most scripts under/usr/share/canit/scripts rather than in /etc/mail/canit.

• CLEANUP: The entire concept of “one-shot messages” has been removed. It was not usefuland served mostly to confuse.

• CLEANUP: “Hit-and-Run” is now consistently referred to as “Greylisting” to keep in line withstandard terminology.

• IMPROVEMENT: The formerly-global ”Auto-populate pending notification addresses” is nowper-stream (therefore per-realm in CanIt-Domain-PRO.)

• IMPROVEMENT: The internal storage-manager code has been changed to make it easier toadd, remove and rename storage-manager nodes.

• MAJOR REORGANIZATION: Most configuration files (mimedefang.conf, db-settings, etc.)have been reorganized into one master configuration file canit.conf.

• BUG FIX: Mail log rotation on Debian-based appliances has been fixed. Previously, logs couldbe rotated twice in quick succession.

• OTHERS: Many other minor bug-fixes and improvements.


313


• UPDATES: Updated ClamAV from 0.93.1 to 0.93.3

• BUG FIX: The global setting ”Silently discard rejected messages rather than remailing withticker” was removed from the Web interface, but not from the filter code. On busy servers, thiscould result in large queues on the ticker host and delays in remailing released messages.

• BUG FIX: A missing JavaScript check on one of the “Reject All as Spam” buttons in the trapdisplay was fixed.

• BUG FIX: The CanIt Domain Routing API call would fail if you supplied only one server fordomain routing. This has been fixed.


• UPDATES: Updated SpamAssassin from 3.2.3 to 3.2.5. Update ClamAV from 0.92.1 to 0.93.1.


Some clamd options have been removed; you MUST remove them from your clamd.conf file orclamd will refuse to start. (CanIt appliances will automatically remove the options.)

The options to remove are:

ArchiveMaxFileSize, ArchiveMaxRecursion, ArchiveMaxFiles, ArchiveMaxCompressionRa-tio and ArchiveBlockMax

• POLICY CHANGE: On new installations, the default for “Tempfail Suspect Messages” is now“Never” rather than “Until-Dispatched”. We decided the change was necessary to avoid am-plification effects on very busy systems and to avoid support queries when people release mailafter several days.

• NEW BINARY PACKAGES: We have added support for Fedora 9 RPMs on i386. We havedropped binary packages for Fedora Core 5 and 6.

• IMPROVEMENT: We have provided a new command-line tool called “canit-api-client”. Youshould begin using it rather than “canit-cmd” because “canit-cmd” will be removed in CanIt4.2.0.

• WORKAROUND: We implemented a workaround for Outlook 2007’s broken form-handlingbehaviour; it lets you accept or reject individual messages from the notification e-mail.

• IMPROVEMENT (CanIt-Domain-PRO only): “Source Address of CanIt Notifications” and“Full name for sender of CanIt notifications” can now be set on a per-realm basis.

• IMPROVEMENT: Global Settings and Stream Settings are now grouped into related sets. Youcan hide the display of sets you’re not interested in seeing.

• IMPROVEMENT: If the DNS lookup to find the RPTN version fails, the System Check pagealerts the administrator.



• IMPROVEMENT: In the trap display, if you have accepted any messages but click “Reject Allas Spam”, CanIt prompts for confirmation first.

• WORKAROUND: If the system time on the ticker is set far in the future and then reset to thecorrect time, ticker tasks may not run. The ticker code now detects clock skew and compensatesfor it.

• IMPROVEMENT/BUG FIX: The API server has had many validation bugs fixed.

• BUG FIX: The PostgreSQL failover module has been update with various bug fixes as well asa workaround for bugs in PostgreSQL 8.3.0 and 8.3.1.

• BUG FIX: If the database is still starting up, the CanIt Storage Manager startup script keepstrying for a while before giving up.

• BUG FIX: A quoting error in decoding certain encoded subject lines has been fixed.

• BUG FIX: The Authentication Mapping web page did not handle a Filter correctly. This is nowfixed.

• BUG FIX: The “Dashboard” Web page did not respect all permissions correctly. This is nowfixed.

• BUG FIX: The ForceToStream attribute is ignored for mail originating from the loopback ad-dress. The old behaviour would sometimes cause released mail to be re-trapped in a differentstream.

• BUG FIX: CanIt would sometimes leave SpamAssassin temporary files littering /tmp; this hasbeen fixed.


• WORKAROUND: “Clickable Webforms” do not work with Outlook 2007 and cannot be madeto work; we added a note to that effect for Outlook 2007 users.

• COSMETIC FIX: The “View System Load” button on the Server Management page was ab-surdly big. It now matches the other buttons.

• BUG FIX: If users defaulted to the “Simplified Interface”, the RSS feed did not work. This isnow fixed.

• BUG FIX: Templates for “Clickable Webform” pending notifications were not being set cor-rectly. This is now fixed.

• BUG FIX: Pending Messages were not being triggered as incidents were created. This is nowfixed.

• BUG FIX (Domain-PRO only): “Clickable Webform” sometimes did not work, depending onwhich realm the user was in. This is now fixed.


315

• BUG FIX: Updating Templates didn’t take effect immediately; this is now fixed.

• BUG FIX: The canit-cmd tool and the API server would not let you set the treat as mx flag forverification servers. This is now fixed.

• BUG FIX: The API server did not work on some versions of Red Hat because Red Hat ships atruly ancient version of Sys::Syslog. We have worked around the problem.

• BUG FIX: We inadvertently packaged the wrong version of the PostgreSQL failover code. Thishas been fixed.


• MAJOR NEW FEATURE: Users can configure an RSS feed of pending incidents. This allowsyou to use your favourite RSS feed reader to monitor your trap.

• MAJOR IMPROVEMENT: Pending notifications are only sent out if there are new pendingmessages. Also, you can configure notifications to be submittable forms; this allows you toaccept or reject messages directly from your e-mail client without having to authenticate withCanIt.

• IMPROVEMENT: You can restrict the days on which Pending Notifications are sent. (You canskip them on weekends, for example.)

• IMPROVEMENT: We use the creation date of an incident when logging it in the statistics tablesrather than the resolution date. However, pending incidents that are resolved some time afterthey are created appear only in daily statistics, not hourly statistics.

• NEW FEATURE: You can set the full name for the source of CanIt notifications.

• IMPROVEMENT: CanIt can be disabled for maintenance from the Web interface. You nolonger have to create /etc/mail/canit/disabled on all machines.

• POLICY CHANGE: We have disabled all spamhaus.org DNS-based RBLs. Spamhaus is be-coming more adamant about enforcing its terms-of-use; if you wish to use Spamhaus-basedtests and do not qualify for free use of the RBLs, please arrange directly with Spamhaus for adata feed contract.

• POLICY CHANGE: We have removed the Sendmail domain-mapping method. The upgradescript replaces it with a Program method for backward-compatibility.

• PERFORMANCE IMPROVEMENT: Marking old incidents as spam has been moved out of thecron job into a ticker task that can operate more leisurely.

• PERFORMANCE IMPROVEMENT: If a stream has too little Bayes training, we don’t use thatstream’s training database. (Before, we’d add the database totals together and if the total waslarge enough, we would use all the data.) This greatly improves performance on sites with manystreams where most of the streams have little Bayes data.



• IMPROVEMENT: Log the IP address of HTTP clients in incident logs.

• IMPROVEMENT: The site administrator can temporarily disable pending notifications.

• SECURITY IMPROVEMENT (CanIt-Domain-PRO only): Only the super-root can create “Pro-gram” or “Program Legacy” user-lookups.

• GUI IMPROVEMENT: The GUI prompts for confirmation before deleting rules, users, etc.

• UPDATES: Update to ClamAV 0.92.1 and Net::DNS 0.63.

• BUG FIX: Fix internal handling of “duplicate key violation” error message from PostgreSQL8.3.

• BUG FIX: Various bugs in the CanIt-API server were fixed.

• BUG FIX: Build problem on NetBSD and FreeBSD was fixed.

• BUG FIX: Under certain conditions, CanIt would break S/MIME signed messages. This hasbeen fixed.

• BUG FIX: Storage-manager startup script incorrectly distinguished upper- and lower-case inhost names. This has been fixed.

• BUG FIX: Prohibit deletion of @@READABLE and @@WRITABLE permission-sets (in baserealm only in CanIt-Domain-PRO.)

• BUG FIX: The PHP code would fail if one Storage Manager node was marked read-only. Thishas been fixed.

• BUG FIX (CanIt-Domain-PRO only): Invalid recipients would always be logged in the “base”realm’s statistics instead of the correct realm. This has been fixed.


• NEW FEATURE: You can enter multiple verification servers (separated by commas) for a givendomain. CanIt tries the servers in order until it receives a definite success or failure indication.

• NEW FEATURE (CanIt-Domain-PRO only): The site administrator can view statistics aggre-gated across all realms.

• CHANGE: We emit key/value logging information when an incident is held or streamed.

• POLICY CHANGE: All of the products (CanIt, CanIt-PRO and CanIt-Domain-PRO) now usethe same filter file.


IF YOU HAVE MODIFIED YOUR FILTER FILE, BE SURE TO TEST YOUR MODIFICA-TIONS WITH THE NEW FILTER BEFORE INSTALLING ON A PRODUCTION SERVER.If you have not modified your filter file, an upgrade will proceed safely.


317

• PERFORMANCE IMPROVEMENT: If you use Storage Manager, the expiry job prunesstorage-manager data in the background.

• BUG FIX (Standard CanIt only): The displayed RPTN statistics would be incorrect if localBayes training occurred. This has been corrected.

• BUG FIX: A PHP warning on the Reports page was corrected.

• BUG FIX: Some minor CSS settings were updated to work better with Internet Explorer.

• BUG FIX: Some minor bugs in the CanIt API and canit-cmd were fixed.

• BUG FIX: Errors in rendering pie charts with very small wedges were corrected.

• BUG FIX: The upgrade code from CanIt to CanIt-PRO would fail to initialize some CanIt-PROtemplates; this has been fixed.


• NEW PACKAGES: We have Debian packages for Etch as well as sarge, and a new Etch-basedISO image.

• NEW FEATURE: The “Stream Settings” page can show where each setting comes from (inother words, which stream it is inherited from.)

• PACKAGING IMPROVEMENTS: Several formerly appliance-only features such as Post-greSQL failover and configuring mail routing from within the CanIt web interface are nowavailable in the RPM versions.

• BUG FIX: The file wal archive command.pl was inadvertently left out of the failover packages;this has been corrected. Additionally, we include a sample failover configuration file.

• BUG FIX: The charts in the Statistics page were adjusted to avoid cutting off Y-axis labels.

• BUG FIX (CanIt-Domain-PRO only): The Known Networks page would insist on an entry inthe force-to-stream column even if you didn’t want one. This has been fixed.

• BUG FIX: Storage Manager would not compile on some old C compilers. This has been fixed.

• BUG FIX: The index on the daily statistics table was suboptimal, leading to slow queries. Thishas been fixed.

• BUG FIX: The Stream Permission page did not expand/contract properly under Internet Ex-plorer. This has been fixed.




• NEW FEATURE: A stream setting can limit the maximum number of entries in the Valid Re-cipients Table. By removing permissions from this entry, a site administrator can limit themaximum number of valid recipients per stream.

• NEW FEATURE: A verification server can listen on a non-standard port; use “servername/port”in the Verification Server table.

• EXPERIMENTAL FEATURE: We calculate Bayes probability based on the Robinson-Fishercalculation. This calculation is not used, but information about it appears in headers and reports.More testing is needed to see if the Robinson-Fisher calculation is actually any better than theNaive Bayes calculation.

• POLICY CHANGE: The main Reports page now shows statistics for all streams if the user hasroot privileges. Unprivileged users only see statistics for their particular stream.

• IMPROVEMENT: The canit-convert-statistics.pl script now works with standard CanIt as wellas CanIt-PRO and CanIt-Domain-PRO.

• BUG FIX: On standard CanIt appliances, we accidentally omitted the PerlLog::Syslog::Abstract module. It is now correctly included.

• BUG FIX: Fixed compilation failure of canit-storage-manager on FreeBSD 5.0.

• BUG FIX (Domain-PRO only): The web interface insists on a fully-qualified stream name forthe “Force-to-Stream” attribute.

• BUG FIX: On *new* installations only, the init-database script did not create /etc/mail/canit/db-settings. This is now fixed.

• BUG FIX: The CanIt REST-based API did not return the same information for list-active-streams as the Web interface. This has been fixed.


• MAJOR NEW FEATURE: Statistics and reporting have been completely reworked. There aremany more reports available and if your PHP installation has the GD extension, you get graph-ical charts.

• MAJOR NEW FEATURE: We have scripts for automatic database failover using PostgreSQL’sPoint-in-Time-Recovery feature. The failover feature is only available on our Debian-basedappliances, however.

• MAJOR ARCHITECTURAL CHANGE: CanIt includes a dedicated “Storage Manager” dae-mon for storing large blocks of textual data. This cluster-aware daemon greatly relieves theload on the PostgreSQL database. It should considerably shrink the size of the database, withattendant improvements in expiry, VACUUM and dump times and overall performance.


319

• MAJOR NEW FEATURE (PRO and Domain-PRO only): The command-line tool has beenreplaced with a completely-new REST-based API. (We ship a replacement command-line toolthat uses the REST API rather than direct database manipulation.)

• NEW FEATURE (Appliance Only): You can specify that the appliance is to treat a name in thedomain routing table as an MX record rather than a host name. This allows for load-balancingback-ends servers using DNS.

• NEW FEATURE: Each stream can request mail to be blind-carbon-copied to an additional e-mail address.

• NEW FEATURE: We have implemented a mechanism similar to RPTN for automatically push-ing out SpamAssassin rules.

• NEW FEATURE: The cron job can be configured to rotate nightly dumps, keeping a config-urable number of nightly dumps.

• NEW FEATURE: “Known Networks” has been enhanced to include a pseudo-network called“SMTP-AUTH”. Settings for that network apply to senders who authenticate using STMPAUTH.

• NEW FEATURE: A ’*’ entry in the domain for a verification server acts as a wildcard. Do notuse this feature if your CanIt server relays outbound mail!

• NEW FEATURE: You can specify how many hours to keep mail in ”Current Statistics.” Thedefault setting of three days is much too long on very busy mail servers.

• UPGRADES: Upgraded bundles software: SpamAssassin from 3.1.8 to 3.2.3; ClamAV from0.90.3 to 0.91.2.

• PACKAGING CHANGES: We have added packages for Fedora 7. We have dropped binarypackages for Solaris, Fedora Core 3 and Fedora Core 4. You will need to install from source onthose platforms.

• GUI IMPROVEMENT: Release notes and all PDF manuals are accessible from the Web inter-face.

• GUI IMPROVEMENT: The “System Check” page has been improved to show the results of allsystem tests. It also shows the currently-loaded RPTN and ruleset versions.

• IMPROVEMENT: We tokenize additional parts of messages for Bayes, such as the sendingrelay and the local and domain parts of the envelope sender.

• UPDATE: Removed “Chickenpox” SpamAssassin rules.

• CHANGE: All relics of the old CanIt-SMB codebase have been removed and consolidated intoCanIt-PRO.

• GUI CHANGE: Web pages are rendered in UTF-8 rather than ISO-8859-1 character set.



• CHANGE: The X-Antispam-Training headers are renamed to X-Antispam-Training-{Forget,Nonspam,Spam}. Some marginal mail clients seem to delete multiple headers withthe same name.

• MINOR IMPROVEMENT (PRO, Domain-PRO only): Administrators can sort incidents bystream when viewing the “*” pseudo-stream.

• GUI IMPROVEMENT: The GUI character set is now UTF-8. This should allow for moreaccurate display of message subjects in non-Western character sets.

• MINOR IMPROVEMENT: The X-Bayes-Prob header now lists which streams’ tokens wereused. This makes it easier to verify that RPTN is being used.

• MINOR IMPROVEMENT: The sample script for synchronizing Bayes databases uses the -Sand -O options with rsync (if your version of rsync supports them.)

• MINOR IMPROVEMENT: The administrator can set the “Forgot your Password?” link fromSetup : Templates.

• RULE CHANGE: Removed the VIRUS WARNING64 SpamAssassin rule which could causefalse-positives.

• BUG FIX (appliances only): You could not edit a domain-routing entry. (You would have todelete/add it). Editing now works properly.

• BUG FIX (PRO, Domain-PRO only): Editing an Address Mapping in the Web interface nowexplicitly clears the “cached” flag.

• BUG FIX: The BccAddress stream permission was not correctly granted by a database upgrade.Now fixed.

• BUG FIX: Obsolete settings are now correctly deleted from the setting desc table upon databaseupgrade.

• BUG FIX: The attachment-stripping code would sometimes log a filename of “unknown” ratherthan the proper filename.

• BUG FIX: The watch-clamd script uses a lock to prevent two concurrent instances.

• BUG FIX: The “Domain Rules” page accepts top-level domains like “jp” or “ca” without com-plaining that they are invalid.

• BUG FIX (Debian appliances only): Ownership and permissions of /var/lib/clamav have beenfixed.

• BUG FIX: The incident creation code has been cleaned up to reduce the chance of race condi-tions.

• BUG FIX (Plain CanIt only): If RPTN downloads are disabled, do not use RPTN data (it isprobably stale anyway.)


321

• BUG FIX: Sendmail accepts addresses like <[email protected]> and<\d\f\[email protected]> equivalently. This can mess up AsIs or ChopDomainstreaming, so CanIt canonicalizes addresses by removing backslashes.

• BUG FIX: The Simplified Interface is disabled if you’re in the “default” stream. (Before, itwould appear but do nothing because default’s inheritance cannot be changed.)

• BUG FIX: In several places in the Web interface, potentially-dangerous actions were performedby a GET request. These have all been converted to POST to make it difficult to accidentallybookmark a dangerous request.

• BUG FIX: The “WHOIS” page would sometimes lose the “s=xxx” parameter in the URL, caus-ing it to refuse to send abuse complaints. This has been fixed.

• BUG FIX: If “One-Shots” are disabled, the Show Active Streams page no longer has a One-Shotcolumn.

• BUG FIX: Generating the “Hit-and-Run” report could consume huge amounts of memory caus-ing PHP to abort. This has been fixed.

• BUG FIX: We test mail against a Verification Server before attempting to stream it. This canreduce load considerably in some configurations.


• ENHANCEMENT: Hit-and-Run detection now takes into account mutating message subjects,making greylisting even more effective.

• UPGRADE: Packaged version of ClamAV has been upgraded to 0.90.3.

• GUI ENHANCEMENT (Appliance Only): The Domain Routing page has a Filter box that letsyou limit which domains are displayed.

• GUI ENHANCEMENT: The message display page makes better use of screen real-estate toreduce back-and-forth scrolling.

• POLICY CHANGE (PRO and Domain-PRO only): We now do greylisting (AKA Hit-and-RunDetection) before streaming. If a message comes in for more than one stream, and *all* of thestreams have greylisting enabled, we greylist the message.

• POLICY CHANGE (PRO and Domain-PRO only): Greylisting applies even to opted-outstreams. (However, since you can now disable greylisting on a per-stream basis, you can turnoff greylisting for opted-out streams if you wish by explicitly disabling greylisting.)

• BUG FIX: The RPTN download task obeys proxy settings in environment variables. (See theLWP::UserAgent man page for details.)





• MINOR ENHANCEMENT: The message display page displays only the main headers (Return-Path, From, To, Subject and Data) by default, with a JavaScript link to reveal/hide all headers.


If you have re-themed CanIt, please test your theme. The base theme now uses an externalJavaScript library rather than emitting inline JavaScript. If your themes inherit from rather thanreplace our themes (the recommended approach), they will most likely work fine.

• BUG FIX (PRO and Domain-PRO only): If an attachment is stripped from an e-mail that hasno text/plain or text/html parts, we add the notice of stripping as a separate text/plain part.

• BUG FIX (PRO and Domain-PRO only): Parameter validation was improved; previously, astream named “0” was not allowed.

• BUG FIX (PRO and Domain-PRO only): If a stream inherited from another stream, the inherit-ing stream’s owner could get spurious Pending Notifications. This has been fixed.




ON SOURCE AND RPM INSTALLATIONS, YOU MAY NEED TO EDIT clamd.conf ANDfreshclam.conf. THE SYNTAX HAS CHANGED. Instead of single words like “LogSys-log”, you must now use “LogSyslog Yes”. PLEASE VERIFY YOUR CLAM CON-FIGURATION FILES AFTER UPGRADING. See http://wiki.clamav.net/Main/UpgradeNotes090 for details

On our Debian-based CanIt appliances, the upgrade script will fix the Clam configuration filesautomatically.

• MINOR CLEANUP: The cron job removes its lock file when it has finished. There is no harmfrom leaving the lock file lying around, but it is cleaner to remove it.

• BUG FIX: All hard-coded colors in the PHP code have been eliminated in favour of CSSstylesheets and classes.

• BUG FIX: A typo in the PHP code could produce a PHP warning; this has been fixed.

• BUG FIX (CanIt-PRO and higher only): Inheritance of the “Use Simplified GUI” preferencewas broken; it is now inherited just like other preferences.

• BUG FIX (CanIt-PRO and higher only): A User Lookup would not accept a comma-separatedlist of servers unless there was whitespace around each comma. This has been fixed so thewhitespace is allowed but not required.


http://wiki.clamav.net/Main/UpgradeNotes090

http://wiki.clamav.net/Main/UpgradeNotes090

323

• BUG FIX (CanIt-PRO and higher only): The “Filter” on the Inheritance page now filters bothby stream and inherited-stream.

• BUG FIX (CanIt-Domain-PRO only): If you deleted a realm mapping, some settings for adomain (Verification Server, Authentication Mappings and Domain Mappings) would be “or-phaned”. Now, they are moved into the new realm that contains the domain.


• NEW ARCHITECTURE: We now have RPM packages for Red Hat Enterprise Linux 5

• BUG FIX: The VACUUM and database backup cron jobs would fail on password- protecteddatabases. They now work correctly.

• BUG FIX: When Bayes votes were acted upon, CanIt forgot to record the vote in the Bayessignature table (although it did correctly update the Bayes statistics.) The vote is now properlyrecorded.

• BUG FIX (CanIt-PRO and higher only): The Web interface and Perl code disagreed about thedefault value for “Permit use of auto-whitelisting”. They are now in agreement.

• BUG FIX: The cron script would sometimes fail to find required Perl modules on Solaris, andthe locking mechanism would fail on Solaris, Both of these problems have been fixed.

• BUG FIX: When switching streams after doing an Advanced Query, the query would be forgot-ten. It is now correctly remembered.

• BUG FIX (CanIt only): Some PHP pages such as Bayes Settings would fail to render on Stan-dard CanIt. This has been fixed.

• BUG FIX: The host IP in the Report pages did not link to a proper WHOIS query URL. Thishas been fixed.


• NEW FEATURE: When you create an LDAP user-lookup, you can specify a connect timeoutfor streaming. (The timeout does not apply to authentication because PHP unfortunately lacksa mechanism to specify the timeout.)

• TRANSLATION IMPROVEMENT: The French and Spanish translations have been updated.

• BUG FIX: The RPM installer correctly recognizes all versions of CentOS 4.

• BUG FIX: Some architecture-specific Debian packages were incorrectly marked “all”. Theyare now marked “i386” as they should have been.

• BUG FIX: Hit-and-run statistics could be incorrect depending on your greylisting settings. Thishas now been fixed.



• BUG FIX: On some versions of DBI and PostgreSQL, the expiry job could fail with a bunch ofSQL syntax errors being emitted. This has been fixed.

• BUG FIX: When verifying RPTN signatures, gpg would sometimes warn about insecure mem-ory. This warning has been suppressed.

• BUG FIX: In the 3.4.x series, whitespace was stripped from the beginning and end of all formentries. If your password began or ended with a space, this would make logging in impossible.3.4.2 no longer strips spaces from password-entry boxes.

• BUG FIX: Entering a blank or non-numeric score for an RBL rule would cause a PHP error.This has been fixed.

• BUG FIX: Upgrading from CanIt to CanIt-PRO could fail when updating the Bayes table. Thishas been fixed.

• BUG FIX: Some missing “NOT NULL” column constraints were added.

• BUG FIX: A useless warning “Non-multipart entity with no bodyhandle??” would sometimesappear in the mail logs; this has been removed.

• BUG FIX: Warnings about undefined variables if a sender is whitelisted due to SMTP AUTHhave been suppressed.


• BUG FIX: Fix a bug in the database upgrade code which could make the schema upgrade fail.

• BUG FIX: Remove a harmless but annoying PHP warning from the trap display.


• MAJOR NEW FEATURE: The old “Access Rights” page is gone. In its place are far moreflexible “Permissions” pages. These allow fine-grained control over permissions on a per-userand per-group basis. The database upgrade code should migrate the old Access Rights to thenew Permissions accurately.

• MAJOR CHANGE (CanIt-PRO and Domain-PRO only): The old “Stream Redirection” conceptis gone. In its place we have “Stream Inheritance”. This is more flexible and simplifies the codea lot. The database upgrade code will migrate Redirection to Inheritance.

• MAJOR IMPROVEMENT (CanIt-PRO and Domain-PRO only): Hit-and-Run (also known as“Greylisting”) can be enabled on a per-stream basis. However, greylisting is now always doneafter the DATA phase of SMTP.

• GUI IMPROVEMENT (CanIt-PRO and Domain-PRO only): When you are viewing the ’*’pseudo-stream, many things are writable (you can dispose of incidents, change rules, etc.)


325

• IMPROVEMENT (CanIt-PRO and Domain-PRO only): The canit-cmd command-line tool hasadditional commands and is fully modular.

• PERFORMANCE IMPROVEMENT: A new caching scheme has reduced the number ofdatabase queries per e-mail substantially, sometimes by more than 50%.

• PERFORMANCE IMPROVEMENT: Many internal code changes and code refactoring havebeen performed to reduce memory usage and improve performance.

• NEW FEATURE: The Verification Servers feature allows you to queue mail (rather than temp-fail it) if the verification server is unreachable. (The default is still to tempfail mail.)

• NEW FEATURE (CanIt-Domain-PRO Appliance Only): Realm administrators can set up mailrouting for domains within their realms.

• NEW FEATURE (CanIt-PRO and higher only): Locked Addresses can be locked to a comma-separated list of domains or addresses, any of which is permitted to send mail to the lockedaddress.

• GUI IMPROVEMENT: (Almost) any GUI page can be set as your default home page.

• GUI IMPROVEMENT: Layout of “Incident Details” page has been made much cleaner.

• IMPROVEMENT: Many more CanIt-generated messages are templatable so you can translatethem or tailor them to fit your site’s policies.

• IMPROVEMENT: The RBL timeout defaults to 7 seconds instead of 30 seconds and is config-urable rather than hard-coded.

• POLICY CHANGE: CanIt’s “System Check” page warns if you do not enable RPTN down-loads.

• POLICY CHANGE: The “Only See Spam” attribute has been removed from the Users table.Instead, use the equivalent Permissions.

• POLICY CHANGE: We have removed support for a shared Bayes database in CanIt-PRO. Itcomplicated the code very much and never really worked well.

• POLICY CHANGE: Obsolete headers X-CanIt-Tag-Reason and X-CanIt-Warning are no longeradded. The information they would have contained is included in the X-Spam-Score header.

• POLICY CHANGE: Source packages no longer ship with Sendmail. You are expected to haveSendmail and Milter installed as prerequisites.

• CHANGE: The canit.cron cron job is now written in Perl rather than Bourne shell.

• BINARY PACKAGES: We have added RPMs for Fedora Core 6 and dropped them for FedoraCore 2.

• IMPROVEMENT: The GUI decodes base64- or quoted-printable-encoded subject lines.



• INCOMPATIBILITY: CanIt now requires PostgreSQL 7.3 or newer. It will NO LONGERWORK with PostgreSQL 7.2.x or older!

• CHANGE: We no longer bundle Crypt-SSLeay with the source installer. You must install oneof Crypt::SSLeay or IO::Socket::SSL as a prerequisite for CanIt.

• IMPROVEMENT: More messages are templatable.

• IMPROVEMENT (Appliance Only): The GUI allows you to remove cluster members if youremove or rename a node.

• IMPROVEMENT: The Advanced Search allows you to search by date range.

• BUG FIX: RPTN would sometimes fail GnuPG signature verification because of bad times-tamps. This has been fixed.

• BUG FIX (CanIt only): RPTN data would not be reflected correctly in the Web interface. Thishas been fixed.

• BUG FIX: All rules that can add to the score (Bayes, SPF, Mismatch) are now reflected in theX-Spam-Score: header.

• BUG FIX: The LDAP user-lookup ignored the “Mail Attribute” setting when authenticating.This has been fixed.

• BUG FIX (CanIt-Domain-PRO only): Realm-mapping lookups use the entire domain, then theparent domain, and so on until a match is found. (For example, “foo.example.com” will searchthe Realm Mapping Table for “foo.example.com”, “example.com” and “com” until it finds amatch.)

• BUG FIX (CanIt-Domain-PRO only): The command-line invocation “canit-cmd del-realmREALMNAME” deletes all traces of a realm (including streams and users.)

• BUG FIX (CanIt-PRO and Domain-PRO): If you require streams to opt-in, then the databaseupgrade script could opt-out the default stream. This has been fixed.


Appendix C

A Testing Topology forCanIt-Domain-PRO

C.1 Introduction

The best way to evaluate CanIt-Domain-PRO is to route real-world mail through it. However, youmay be hesitant to place CanIt-Domain-PRO in production without testing it first. So we’ll show youhow to set up CanIt-Domain-PRO for test purposes, and then how to put it into production in a safeway. The test topology makes it very easy to back out of CanIt-Domain-PRO if you decide to do so.

C.2 Assumptions

We make the following assumptions about your current e-mail setup:

• You already have a mail server that is your primary MX record, and you control that server andits network. The existing mail server may run Sendmail, but it doesn’t have to—it could runNetscape Messenger, Microsoft Exchange, or any other mail server software of your choice.

• You have a spare Intel-architecture server for installing Linux and CanIt-Domain-PRO. Thisserver should have sufficient horsepower to handle all of the mail for your domain or domains.While you can use other supported UNIX operating systems for CanIt-Domain-PRO, the in-structions in this paper are specific to Linux. If you are an experienced UNIX and Sendmailsystem administrator, you can probably translate them for your own system.

• You control your DNS settings and can publish MX records for your domains.

C.3 Network Setup

Figure C.1 illustrates the assumed existing network setup followed by the new network setup. Notethat your actual setup may be more complex and may include firewalls, demilitarized zones, etc.


328 APPENDIX C. A TESTING TOPOLOGY FOR CANIT-DOMAIN-PRO

Conceptually, however, we assume you have an existing mail server which is the primary MX machinefor your domains, and which is connected to the Internet.

The test network shows how the CanIt-Domain-PRO server is configured to accept mail from theInternet and relay it to your actual mail server.

Internet Existing Mail Server

Internet Existing Mail ServerCanIt Server

Primary MXSecondary MX

Original Network

Test Network

Figure C.1: Network Configurations

C.4 Build the CanIt-Domain-PRO Server

To build the CanIt-Domain-PRO server, install Linux on an Intel Architecture server. Be sure to installApache, PHP and PostgreSQL, which are included with most Linux distributions. Alternatively, installour Debian-based appliance build.

C.5 Configure the CanIt-Domain-PRO Server to Relay Mail

You’ll need to edit two files on the CanIt-Domain-PRO server to configure Sendmail to relaymail. Make a list of all the domains for which your existing mail server accepts mail. Let’ssuppose you own the domains example1.com and example2.net, and accept mail for both onthe machine mail.example1.com. Finally, we’ll assume the CanIt-Domain-PRO server is calledcanit.example1.com.


C.6. ROUTE TEST MAIL 329

C.5.1 Enable Relaying

First, you must enable relaying for the domains you control. To do this, edit the file/etc/mail/access and add a line for each domain, something like this:

To:domainname.tld RELAY

In our example, we’d add two lines to /etc/mail/access:

To:example1.com RELAYTo:example2.net RELAY

C.5.2 Configure Forwarding Relays

Next, you have to tell CanIt-Domain-PRO where to relay mail for the domains. Edit the file/etc/mail/mailertable and add a line for each domain, something like this:

domainname.tld esmtp:[relay.domainname.tld]

In our example, recall that mail.example1.com handles mail for both domains, so our mailertablewould look like this:

example1.com esmtp:[mail.example1.com]example2.net esmtp:[mail.example1.com]

C.5.3 Rebuild Sendmail Databases

Finally, you need to rebuild Sendmail’s internal databases to reflect these changes. Simply execute thefollowing Linux commands as root:

cd /etc/mailmake

C.6 Route Test Mail

Up until this point, your existing mail server has continued to act as it always does. The CanIt-Domain-PRO machine, although “live” and on the network, is not handling any mail traffic. Nowcomes the time to route mail through the CanIt-Domain-PRO server. There are two options to routetest mail through the CanIt-Domain-PRO server:



C.6.1 Direct Injection

The least disruptive method is to directly inject test messages into the CanIt-Domain-PRO server. Runan SMTP client and send messages via the CanIt-Domain-PRO server. Verify that they are receivedand that spam messages are held.

You can use an e-mail client such as Mozilla or Microsoft Outlook for testing purposes. Simply setthe outgoing SMTP machine to be the CanIt-Domain-PRO relay (in our example, canit.example1.comand send messages to people in your organization.

Alternatively, you can use a UNIX or Linux machine with its own DNS server. Create an MX recordfor your domain pointing to the CanIt-Domain-PRO server and send messages. Remember, only thetest machine thinks that CanIt-Domain-PRO is your mail relay; the rest of the Internet still uses yourexisting mail server.

C.6.2 Create a Test Subdomain

Another option is to create a test subdomain, such as test.example1.com. Configure your regu-lar mail server to accept mail for that domain, and don’t forget to modify the CanIt-Domain-PROserver’s access and mailertable files to relay mail for that domain. Then publish an MX record fortest.example1.com pointing to canit.example1.com. You can then send mail from anywhere in the In-ternet to someone at test.example1.com and it will be relayed through the CanIt-Domain-PRO server.Existing mail to your proper domain, however, will still travel via your old mail server.

C.7 Route Real Mail

Once CanIt-Domain-PRO has passed the initial tests, it’s time to route real e-mail through it. Thesafest way to do this is to add an additional MX record for your domains. This record should have thehighest priority, and point to the CanIt-Domain-PRO server.

For example, let’s suppose your existing MX records look like this:

example1.com. 1d IN MX 10 mail.example1.com.example1.com. 1d IN MX 15 m2.example1.com.

Simply add another MX record like this:

example1.com. 1d IN MX 5 canit.example1.com.

and propagate the DNS changes. Mail for your domain will now be routed through the CanIt-Domain-PRO machine. In an emergency, if you need to take the CanIt-Domain-PRO machine offline, simplykill Sendmail on the CanIt-Domain-PRO server. Relays attempting to deliver mail to your domainwill first try the CanIt-Domain-PRO server and immediately get a “Connection refused” error. Theywill fall back very quickly to the remaining MX records, and mail will flow as usual.


C.8. OUTGOING MAIL 331

Note: This test setup is not a viable topology for stopping spam. Because CanIt-Domain-PRO sendstemporary-failure codes for suspect mail, if your real mail server has an MX record, the sender willsimply relay the spam directly to it. For production use, all of your public records should either:

• Be running CanIt-Domain-PRO, or

• Relay to a machine running CanIt-Domain-PRO.

The actual internal mail server should be hidden (no MX record) and ideally firewalled off, so onlythe CanIt-Domain-PRO relay can connect to it.

C.8 Outgoing Mail

If you want to pass outgoing mail through CanIt-Domain-PRO, configure your mail server to usethe CanIt-Domain-PRO server as a “SmartHost”. This is a host to which all non-local mail will besent. The details of SmartHost configuration differ among mail servers; consult your mail serverdocumentation for details.


Appendix D

CanIt-Domain-PRO Architecture

D.1 Introduction

CanIt-Domain-PRO is based on the Sendmail Milter API, described at http://www.milter.org/developers/design. Milter is a scalable API for doing site-wide filtering of e-mail.

Figure D.1 shows how CanIt-Domain-PRO interfaces with Milter.


http://www.milter.org/developers/design

http://www.milter.org/developers/design

334 APPENDIX D. CANIT-DOMAIN-PRO ARCHITECTURE

Sendmail Sendmail Sendmail

mimedefang−multiplexor

mimedefang

mimedefang.plmimedefang.pl mimedefang.pl

Milter Interface

Unix−domain Socket

Pipes

Figure D.1: CanIt-Domain-PRO Architecture

D.2 CanIt-Domain-PRO Architecture

In Figure D.1, we show multiple sendmail processes communicating with a single mimedefangprocess. The mimedefang executable uses the Milter reference library, and is therefore multi-threaded. The mimedefang process is shown in cyan because it is the only multi-threaded pro-cess in CanIt-Domain-PRO; all others are single-threaded. The interface between mimedefang andsendmail may be a local (UNIX-domain) socket or a TCP socket.

mimedefang takes care of accepting e-mail headers and bodies from sendmail and writing themto a temporary spool directory (typically, /var/spool/MIMEDefang). It then sends short com-mands to mimedefang-multiplexor.

mimedefang-multiplexor listens on a UNIX-domain socket and manages a pool of Perl pro-cesses which do the actual filtering. The multiplexor has the following responsibilities:

1. It listens for requests from mimedefang and assigns them to one of the Perl processes.

2. It starts more Perl processes (up to a configured limit) if load increases.


D.3. STARTING AND STOPPING CANIT-DOMAIN-PRO 335

3. During times of low load, it kills off Perl processes (down to a configured limit.)

4. It kills Perl processes which have processed a configured number of messages. This is done toavoid potential memory leaks.

5. It kills Perl processes which take too long to scan a message or which stop responding to re-quests.

mimedefang.pl is the actual Perl filtering program. It listens for requests (from the multiplexor)on its standard input, and writes results to its standard output. The commands and results exchangedare quite short; any modifications to the e-mail message are done in the spool directory.

Because the multiplexor manages several Perl processes, the Perl filters do not have to be thread-safe.In addition, the “pool-of-preforked-processes” architecture scales very well on SMP systems, and isefficient, robust and reliable.

D.3 Starting and Stopping CanIt-Domain-PRO

CanIt-Domain-PRO is started by a script called /usr/share/canit/scripts/canit-system.This script handles the starting and stopping of multiple CanIt-Domain-PRO services and is invokedwith a single argument; possible arguments are:

start Starts all relevant CanIt-Domain-PRO services on this host.

stop Stops all CanIt-Domain-PRO services running on this host.

stop-most Stops all CanIt-Domain-PRO services running on this host except for those servicesthat are required for database access.

restart Equivalent to stop followed by start.

stop-gracefully Stops all CanIt-Domain-PRO services running on this host. Unlike stop, thisargument waits for any processes performing critical work to exit on their own. It is thus saferstop, but may take a lot longer.

stop-most-gracefully Similar to stop-most but waits for processes performing criticalwork to exit on their own.

restart-gracefully Similar to restart but waits for processes performing critical work toexit on their own.

check Starts all CanIt-Domain-PRO services that should be running on this host but are not, andstops all services that are running on this host but should not be.

status Prints the status of CanIt-Domain-PRO services on this host. Exits with an exit code of 0if all services that should be running are running, and all services that should be stopped arestopped. Exits with a code of 1 otherwise.



D.4 Static Configuration Files

Most CanIt-Domain-PRO services read a configuration file called/etc/mail/canit/canit.conf for static configuration settings, before reading the re-mainder of the configuration from the database. This file contains local configuration items thatdiffer from factory defaults. The meanings of some of the configuration settings are described below.Boolean variables can take the values yes or no, while other variables are integers or strings.

D.4.1 Database Settings

The following settings exist in the [database] section of the configuration file.

db host (string) should be set to the host name or IP address of the database server. If the databaseserver is on this host, this setting should be blank.

db name (string) should be set to the name of the CanIt-Domain-PRO database, typically spam.

db super (string) should be set to the name of the PostgreSQL super-user, typically postgres.

db user (string) should be set to the name of the PostgreSQL user for normal database access, typi-cally spam.

db super passwd (string) should be set to the name of the PostgreSQL super-user’s password, if youare using MD5 authentication.

db passwd (string) should be set to the name of the PostgreSQL normal user’s password, if you areusing MD5 authentication.

db port (integer) may be set to the TCP port number of the PostgreSQL server. You should set thisonly if your PostgreSQL server listens on a non-standard port.

db connect timeout (integer) should be set to the timeout for connecting to the PostgreSQL databasein seconds. The default timeout is 20 seconds.

D.4.2 Cron Settings

The [cron] section contains several settings that control the nightly cron job. They are:

compress dump (boolean) If set to yes, then the nightly database dump will be compressed withgzip.

skip dump logindex tables If set to yes (the default), then the nightly database dump will not in-clude the log-indexer tables. They can be reconstructed from log files if necessary. However, ifyou want them dumped, set this parameter to no. Note that this parameter has no effect if thelog-indexing CanIt-Domain-PRO component is not installed.

keep dumps (integer) Specifies how many nightly dumps to keep. CanIt-Domain-PRO will rotatethe nightly dumps, keeping at least keep dumps days’ worth of dumps.


D.4. STATIC CONFIGURATION FILES 337

bayes data expiry days (integer) Specifies when Bayes databases should be deleted. If a Bayesdatabase has not been modified in this many days, it is considered stale and removed. Thedefault value is 365. Any value less than 90 is ignored and silently changed to 90.

seen address expiry days (integer) Specifies how long CanIt-Domain-PRO should track valid ad-dresses that have been seen. The default value is 61; acceptable values range from 30 to 365.

parallel tasks (integer) One of the tasks the cron job does is to check which domains validaterecipients. Normally, this is done by a single process, which can be slow. You can setparallel tasks to a number from 1 to 500 to specify how many parallel tasks to run todo recipient verification. Note that you should not set this value to more than about half ofmx maximum in the [mimedefang] section (described below).

autotask billing push days If set to a comma-separated list of numbers, the Autotask billing updatecron job is only run on the corresponding days of the month. For example, setting this variableto 1,15 would run the Autotask update on the 1st and 15th of the month.

connectwise billing push days If set to a comma-separated list of numbers, the ConnectWise billingupdate cron job is only run on the corresponding days of the month. For example, setting thisvariable to 1,15 would run the ConnectWise update on the 1st and 15th of the month.

D.4.3 MIMEDefang Settings

These settings exist in the [mimedefang] section of the configuration file. As most do notneed to be modified from factory defaults, you may not have a [mimedefang] section in your/etc/mail/canit/canit.conf file, so do not be alarmed if it does not exist.

mx user (string) should be set to the user ID of the mimedefang processors. This should nearlyalways be a dedicated user called defang.

mx relay check (boolean) enables filtering of relay IP addresses during SMTP connection. CanIt-Domain-PRO does not filter at connect-time, so this should be set to no.

mx sender check (boolean) enables checking of the sender address in the SMTP “MAIL FROM:”command. CanIt-Domain-PRO performs sender checks at “RCPT TO:” time in order to takerecipient streams into account, so this should always be set to no.

mx recipient check (boolean) enables checking of the recipient address in the SMTP “RCPT TO:”command. CanIt-Domain-PRO requires this check, so it must be set to yes.

mx log (boolean) enables logging. This should always be set to yes.

mx requests (integer) specifies how many requests each Perl worker will handle before being killed.The filters are killed after this number of requests to eliminate any possibility of problems dueto memory leaks. The default is 200, which should be reasonable for most installations.

mx lifetime (integer) specifies how many seconds each Perl worker is allowed to live before beingkilled. The default setting of -1 means not lifetime limit is imposed, and this is the recommendedvalue.



mx max recipok per domain (integer) specifies how many concurrent “RCPT TO:” checks are al-lowed per domain. The default is zero, which means no limit. You can set this to some numbersmaller than mx maximum to reduce the likelihood of one domain affecting others adversely.For example, if one domain has a slow or dead verification server, limiting this setting protectsother domains from denial of service caused by all scanners doing RCPT TO: checks for theslow domain.

mx minimum (integer) specifies the minimum number of Perl filters to keep running, even if thesystem is idle.

mx maximum (integer) specifies the maximum number of Perl filters to run concurrently, no matterhow busy the system is.

Note that each Perl filter requires a database connection. The default installation of PostgreSQLpermits only 32 simultaneous database connections. If you need more than this, you shouldincrease the number of PostgreSQL back-ends with the “-N” and “-B” postmaster optionswhen you start the database. Please see the postmaster(1) and pg ctl(1) man pages for details.

mx idle (integer) specifies how long in seconds a Perl process should be idle before it is killed off. Af-ter a period of heavy load, idle processes eventually get killed off until there are mx minimumPerl filters running.

mx busy (integer) specifies how long in seconds a Perl filter is allowed to process a message. Ifthe filter takes longer than this, it assumed to have hung up and is killed, and the message istempfailed.

mx cmd timeout (integer) specifies how long in seconds to wait for commands and results to betransferred between mimedefang and mimedefang-multiplexor.

mx worker delay (integer) specifies how long to wait after starting each Perl filter. If the systemis idle, but fewer than the minimum number of filters are running, a new filter is started eachmx worker delay seconds.

mx min worker delay (integer) specifies that the multiplexor must not start workers more quicklythan the specified delay, no matter what. Even if the system is busy, a new filter will not bestarted more often than every mx min worker delay seconds. Setting this to 1 or 2 secondsmay help your machine withstand a sudden surge in e-mail; it helps smooth out sudden loadincreases. However, it may cause delays as some mail is tempfailed.

mx max rss (integer) specifies the maximum resident-set size in kB of each Perl filter process. Onsystems which support this limit, a Perl filter which exceeds this limit is killed. If set to zero,the limit is ignored.

mx max as (integer) specifies the maximum virtual address space in kB of each Perl filter process.On systems which support this limit, a Perl filter which exceeds this limit is killed. If set to zero,the limit is ignored.

mx stats (boolean) specifies that the multiplexor should log statistical information in/var/log/mimedefang/stats.



mx flush stats (boolean) specifies that the multiplexor should flush/var/log/mimedefang/stats each time it writes a line to the file.

mx stats syslog (boolean) specifies that the multiplexor should log statistical information using sys-log.

mx socket (string) specifies the full path to the UNIX-domain socket used for communicationbetween mimedefang and mimedefang-multiplexor. For CanIt-Domain-PRO, thisshould not be changed.

group accessible files (boolean) specifies whether or not the files and sockets created by MIMEDe-fang should be group-accessible. If you set this to yes, then MIMEDefang-created files aregroup-readable and sockets are group-readable and group-writable.

log times to syslog (boolean) specifies whether or not to log filter times using syslog. If you set thisto yes, then CanIt-Domain-PRO will log lines similar to this in your mail log:

gBNEeeI9004056: Filter time is 231ms

syslog ident (string) specifies the identifier to include in syslog messages. It defaults to CanIt. Youshould not change this; it will be used by future versions of CanIt-Domain-PRO for log analysis.

mx embed perl (boolean) specifies whether or not the multiplexor should use an embedded Perlinterpreter. Normally, when a Perl worker is needed, the multiplexor forks and the childexecs a Perl program. If you set this to yes, then the multiplexor uses an embedded Perlinterpreter that reads the Perl filters only once. When a new worker is needed, only a fork isdone. The overhead of the exec and the Perl interpreter initialization is avoided.

On some systems, it is not possible to embed a Perl interpreter. If you set this flag to yes onsuch a system, a warning is logged to syslog and CanIt-Domain-PRO continues as if the flagwere no.

On some systems, it is possible to embed a Perl interpreter, but not to safely destroy it and createanother interpreter in the same process. On such systems, a warning is logged if you force afilter reread. This will not affect the operation of CanIt-Domain-PRO, but if you edit the actualPerl filter file, you will need to do a (more expensive) mimedefang-ctrl restart ratherthan the cheaper mimedefang-ctrl reread.

D.4.4 Filter Settings

A few filter settings are stored in the configuration file rather than in the PostgreSQL database. Theyreside in the [filter] section. These filtering settings are used in the event that the database is notavailable.

admin address The e-mail address of the CanIt-Domain-PRO administrator.

database down action This setting can take one of three values:



• tempfail (the default). CanIt-Domain-PRO will tempfail mail if the PostgreSQLdatabase server is non-responsive.

• accept. If the database is down, mail will be delivered un-scanned with a warning addedin the X-Spam-Score: header.

• minimalfilter. CanIt-Domain-PRO will run with bare SpamAssassin rules in tag-only mode.

database down virus action If database down action is set to ’accept’, this parameter con-trols how viruses are handled while the database is down. It must be set to one of reject, discardor accept.

database down tag score If database down action is set to ’minimalfilter’, this parametercontrols the threshold at which incoming mail is tagged. The default value is 5.

database down reject score If database down action is set to ’minimalfilter’, this parametercontrols the threshold at which incoming mail is auto-rejected. The default value is 20.

max helper program vsz CanIt-Domain-PRO sometimes calls helper programs as part of filtering(for example, to list the contents of ZIP archives.) This parameter controls the memory limitfor helper programs in kilobytes. The default value is 1048576, or a 1GB memory limit. We donot recommend lowering the limit below about 200MB or 204800. If you set this parameter tozero, then no limit is imposed. We do not recommend this as a bug in a helper program maylead to a denial of service unless its memory is limited.

D.4.5 Ticker Settings

The [ticker] section contains settings related to the ticker tasks run by the CanIt daemon. If youwish to change the settings, be sure to change them on the ticker machine (or on all machines for easeof maintenance.)

pending notifications parallel senders (integer). Normally, CanIt-Domain-PRO uses one processto send pending notifications. This can take a long time; you can run multiple parallel pro-cesses to speed it up. This parameter can range from 1 to 100, though the maximum value werecommend is about 30.

pending notifications throttle db queries (boolean, default false.) If this is set to true, then thepending notification tasks sleep if a particularly expensive query to gather pending messagestake a long time. This sleep time is dynamically adjusted (see the next setting.) Note that thissetting and the next are ignored unless pending notifications parallel senders is greater thanone.

pending notifications throttle target duration (integer, default 4). This specifies the target dura-tion in seconds of the pending-notification SQL query. If the query takes longer, then the sleeptime between notifications is increased. If it takes less time, then the sleep time is decreased.



pending notifications randomize order (boolean, default false.) If this is set to true, then CanIt-Domain-PRO randomizes the order in which Pending Notifications are sent. By default, CanIt-Domain-PRO sorts them in realm/stream order. If you have large realms with many users,randomizing the order can help lower the load on any particular back-end mail server.

remail held messages parallel senders (integer). Normally, CanIt-Domain-PRO uses a single pro-cess to release and remail held messages. You can specify up to 50 parallel processes if youfind that releasing held messages is taking too long. Note that the number you specify here isan upper limit. If there are not many held messages requiring release, CanIt-Domain-PRO mayuse fewer processes than this limit.

index archived mail parallel indexers (integer). This setting applies only if you have installed theCanIt Archiver add-on. Normally, CanIt-Domain-PRO uses only a single background processto index mail that has been archived. If this process is unable to keep up with your archivingvolume, you may specify up to 50 parallel indexing processes.

D.4.6 Cluster Communication Settings

The [remote] section contains settings that control intra-cluster communication. CanIt-Domain-PRO uses SSH for a variety of cluster-management tasks; the available settings are:

ssh address family (string) Can be set to one of inet to force intra-cluster communication to useIPv4; inet6 to force it to use IPv6 or any to have SSH use the default strategy of trying IPv6first and then falling back to IPv4.

D.4.7 Storage Manager Settings

The [storagemanager] section contains the following settings related to Storage Manager:

pidfile (string) A file used by the Storage Manager server to write its process ID andto lock against concurrent Storage Managers. The default value is /var/run/canit-storage-manager.pid.

rootdir (string) The root directory under which data are stored. The default value is /var/lib/canit-storage-manager.

archive root (string) The root directory under which mail is archived (used only if you have installedthe Archiver component.) The default is /var/lib/canit/mail-archive.

listen backlog (integer) The value of the “backlog” parameter for the Storage Manager daemon’slisten() system call. The value can range from 5 to 128; the default is 16.

user (string) The UNIX user as which the Storage Manager server should run. The default value isdefang.

client retry delay (integer) specifies the delay in reconnecting to a dead storage manager node. Ifa CanIt-Domain-PRO cluster node fails to connect to a storage manager node, it will not retry



the connection for client retry delay seconds. This can help prevent a dead storagemanager node from bogging down the clients in blocked connect calls.

client connect timeout (integer) specifies the timeout in seconds for a connection attempt to a Stor-age Manager node. The default is 5 seconds.

client operation timeout (integer) specifies the timeout in seconds for a read or write operation to aStorage Manager node once connection has been established. The default is 20 seconds.

order (string) specifies the order in which to try Storage Manager nodes. The default is “auto”,in which case CanIt-Domain-PRO periodically measures the latency to each Storage Managernode and accesses them in order of increasing latency (fastest to slowest). If you want to specifya particular order, set the value to a space-separated list of fully-qualified host names. The hostswill be tried in the order given. If you do not specify all the hosts, then any remaining hosts aretried after the ones specified by the order parameter.

check latency interval (integer) specifies how often in seconds to measure Storage Manager latencyif the order is set to “auto”. This value can range from 300 to 86400 seconds, with the defaultbeing 3600 seconds or one hour.

disk bandwidth percent for pack prune (integer) specifies how much of the available disk band-width the Storage Manager should use when pruning very old files and packing old filesinto CDB containers. By default, the nightly maintenance task that prunes and packs oldfiles will use all available disk bandwidth. If you find this increases the load too much, setdisk bandwidth percent for pack prune to an integer from 25 to 100. A setting of 50, forexample, will make the Storage Manager sleep after each disk operation for as long as the diskoperation took, meaning a 50-50 split between disk operations and sleeps.

D.4.8 Maintenance Notification

If CanIt-Domain-PRO is unable to connect to the database, the Web interface normally prints anerror message. If you create a file called /etc/mail/canit/db-error.html that containsHTML text, the contents of that file are sent to the browser instead, in a page entitled “System Downfor Maintenance”. Thus, if you know you’ll be bringing the system down, create an appropriatedb-error.html file and stop PostgreSQL (or firewall it off from the Web server.) Be sure to deletedb-error.html once the maintenance has been completed.

Rather than stopping PostgreSQL, you can also create a file called /etc/mail/canit/IN\_MAINTENANCE\_MODE. If this file and /etc/mail/canit/db-error.html are both read-able, then CanIt-Domain-PRO puts the Web interface into maintenance mode.

D.5 Tuning CanIt-Domain-PRO

Tuning CanIt-Domain-PRO is a bit like tuning Sendmail: A black art. Nevertheless, we can offersome guidelines which should help improve the performance of your CanIt-Domain-PRO installation.


D.5. TUNING CANIT-DOMAIN-PRO 343

D.5.1 Memory

Your CanIt-Domain-PRO server should have sufficient memory. As a rule of thumb, you should haveabout 50MB of memory for each concurrent Perl filter. If you set the maximum number of Perl filtersto 16, for example, your machine should have at least 800MB of physical memory.

Your CanIt-Domain-PRO server should also have sufficient swap space that a sudden flood of e-maildoes not cause exhaustion of virtual memory. An additional 32MB of swap space for each Perl filteris probably a good rule of thumb.

D.5.2 Disk

You should have fast, reliable disks on your CanIt-Domain-PRO server. In particular, the CanIt-Domain-PRO spool directory (/var/spool/MIMEDefang) is heavily used, and it may be worthputting it on its own disk. Even better, put the spool directory on a RAM disk, assuming you havesufficient memory. A RAM-based CanIt-Domain-PRO spool directory is a large win, especially onsystems like Solaris with relatively conservative file systems.

To calculate the amount of RAM you’ll need for the spool, multiply the size of the largest messageyou’ll accept by the maximum number of concurrent filters, and then multiply by 3 as a safety factorfor CanIt-Domain-PRO processing. For example, if you accept messages up to 3MB, and you’ll haveat most 8 Perl filters running, then your /var/spool/MIMEDefang space should be at least 72MB.If you use a RAM disk for the spool directory, add this memory to the memory requirements in theprevious section.

D.5.3 Solaris-Specific tmpfs Note

Solaris is very conservative about committing writes to disk. On a busy Solaris server, consider itmandatory to put /var/spool/MIMEDefang on a RAM-based tmpfs file system. The perfor-mance improvement will be dramatic.

D.5.4 CPU

Spam-scanning is quite CPU-intensive, but in modern computers, the CPU is unlikely to be the bot-tleneck. If the CPU does prove to be a bottleneck, you should consider a faster machine, or even amultiprocessor machine.

D.5.5 Sendmail

Tuning Sendmail is quite complex; for a review of some of the issues involved, we recommend “Send-mail Performance Tuning” by Nick Christenson, Addison-Wesley, ISBN 0-321-11570-8.



D.6 Dealing with Overload

Normally, the resources which first become overloaded in a mail server are disk or network bandwidth.However, a server with CanIt-Domain-PRO installed is more likely to run out of CPU power or mem-ory, simply because content-scanning is relatively expensive. If your CanIt-Domain-PRO machinebecomes overloaded to the point that very little mail is flowing and the machine is struggling, here aretuning tips to help you recover.

D.6.1 Tune CanIt-Domain-PRO and Sendmail

In addition to the tuning tips in Section D.5, two parameters are particularly helpful in letting the CanIt-Domain-PRO server deal with overload: In /etc/mail/canit/canit.conf, set mx maximumin the [mimedefang] section to a fairly low number, around 5 or 6. On most hardware, this shouldlimit the impact of scanning on CPU and memory. It will allow the CanIt-Domain-PRO machine toprocess incoming mail smoothly until the overload conditions abate.

In conjunction with mx maximum, it is very useful to set Sendmail’s ConnectionRateThrottleoption. If you set this to 3, for example, Sendmail will accept at most 3 SMTP connections per second.Again, this lets your machine process mail smoothly until overload conditions abate.

So if your server becomes overloaded, follow these recovery steps:

• Set mx maximum to 5, and ConnectionRateThrottle to 3. (If you useM4 to generate the sendmail configuration file, the M4 parameter is calledconfCONNECTION RATE THROTTLE.)

• Watch the load carefully. If your machine appears to have idle time and free memory on itshands, cautiously increase the parameters until throughput seems to be maximized.

D.6.2 Network Architecture

A good way to deal with temporary overload conditions is to have a secondary MX machine thatsimply relays mail without doing any scanning. It will queue messages that the primary machinecannot handle, and then deliver them serially to the primary machine, smoothing out the load. Thedisadvantage of this scheme is that some relay-IP tests do not work as effectively, and the secondaryMX machine may have to generate bounce messages.

If your CanIt-Domain-PRO machine is overloaded a lot of the time, we suggest setting up a sec-ond equal-weighted MX machine with CanIt-Domain-PRO installed. The two CanIt-Domain-PROmachines can share the same PostgreSQL database, since database access is rarely the bottleneck.Having two equal-weighted MX records will spread the load over both machines.


Appendix E

CanIt-Domain-PRO HOWTOS

E.1 Restoring a Database from a Dump

The CanIt-Domain-PRO cron job makes a text dump of the entire database every night; the databaseis dumped into /var/spool/Canit-Spam-DB-Backup/SPAM-DATABASE-BACKUP. Youshould back this file up to ensure the integrity of your spam database.

If, for some reason, you need to restore the database from the text file, follow this procedure. Note thatyou may need to supply the full path to the PostgreSQL utilities like pg dump, psql, createuser,etc.

All of these examples assume that the PostgreSQL superuser is named postgres. This is likely tobe true on Linux and Solaris, but some platforms use pgsql instead (this is the setting in FreeBSD’sport of PostgreSQL.)

1. Stop CanIt-Domain-PRO, the ticker, Sendmail and the CanIt-Domain-PRO Web interface.

2. Dump your existing database, just to be safe. Be sure to do this in a directory with sufficientspace:

$ pg dump -U postgres spam > spam-dump-file.txt

3. Drop the database:

$ dropdb -U postgres spam

4. Create an empty database:

$ createdb -U postgres -E sql-ascii -l C -T template0 spam

5. Restore the database contents from the nightly dump file:

$ psql -U postgres -d spam < SPAM-DATABASE-BACKUP

6. Analyze the database to update statistics for the query optimizer:

$ psql -U postgres -d spam -c ’ANALYZE VERBOSE’

Do not omit the ANALYZE step or your database will be very slow.


346 APPENDIX E. CANIT-DOMAIN-PRO HOWTOS

7. Restart CanIt-Domain-PRO, Sendmail and the CanIt-Domain-PRO Web interface.

Note: The steps above apply if you restore the database onto the same machine it was originally on. If, forsome reason, you had to completely rebuild the machine, follow the steps in Section E.5.

E.2 Firewall Settings

Many people run CanIt-Domain-PRO behind a packet-filtering firewall. If you do, be sure to permitaccess to the following ports. “Inbound” and “Outbound” are from the perspective of the CanIt-Domain-PRO machine. For example, if we say that outbound TCP port 80 must be open, we meanCanIt-Domain-PRO must be able to initiate TCP connections to an external machine with a destinationof port 80. And when we say inbound TCP port 25 must be open, we mean CanIt-Domain-PRO mustbe able to accept TCP connections from another machine destined to port 25.

E.2.1 Firewall Rules: External Hosts

CanIt-Domain-PRO needs the following ports open for communication with external machines on theInternet:

• Inbound and outbound TCP port 25 for SMTP.

• Inbound TCP port 22 for SSH access.

• Outbound TCP and UDP port 53 for DNS lookups

• Outbound TCP port 80 and port 443 for software updates, RPTN downloads and ClamAV sig-nature downloads.

• Outbound TCP port 873 for rsync access to additional ClamAV signatures.

• Outbound UDP port 6568 to report IP address reputation data back to Roaring Penguin Soft-ware.

• Possibly inbound and outbound UDP port 123 if you are using NTP to synchronize the clock.

E.2.2 Firewall Rules: Internal Hosts

CanIt-Domain-PRO needs the following ports open for communication with internal machines in yourorganization. CanIt-Domain-PRO always needs port 25 open; the other three items are required onlyif you use the corresponding User Lookup (Chapter 7.)

• Outbound TCP port 389 for LDAP lookups and/or 636 for LDAPS lookups.

• Outbound TCP port 110 for POP3 lookups and/or 995 for POP3S lookups.

• Outbound TCP port 143 for IMAP lookups and/or 993 for IMAPS lookups.

• Outbound TCP port 25 for verification servers and e-mail delivery.


E.3. RUNNING SOMETHING AFTER THE NIGHTLY CRON JOB COMPLETES 347

E.2.3 Firewall Rules: Intra-Cluster Hosts

If you have a cluster of CanIt-Domain-PRO machines, the following ports are used between clustermembers and should be open:

• TCP port 5432 (typically) for PostgreSQL database connections.

• TCP port 6568 (typically) for Storage Manager connections.

• TCP port 22 for SSH connections.

Note that the PostgreSQL and Storage Manager ports must be firewalled off from external hosts andhosts that are not members of the CanIt-Domain-PRO cluster. Additionally, if you use pgbouncer,you must firewall off its port (6432/tcp by default) from external hosts and/or configure pgbouncerto listen only on the loopback address 127.0.0.1.

Note: An attacker who gains access to the PostgreSQL database can do an enormous amount of damage,from compromising LDAP passwords to reading and altering mail all the way up to potentially gain-ing root access on all cluster members. For that reason, it is absolutely critical to protect the databasefrom unauthorized users. In particular, do not create local UNIX users on any CanIt node unless thoseusers are trusted and authorized to administer the CanIt cluster with root-equivalent privileges.

E.3 Running Something after the Nightly Cron Job Completes

The script /usr/share/canit/scripts/canit.cron runs once a night to perform variousmaintenance tasks. (Note that on some systems, canit.cron might be located in a different direc-tory.)

If a file called post-cron-hook is present in the same directory as canit.cron and is exe-cutable, then it will be run as root after all other cron tasks have been completed. You can use thisscript for whatever purposes you like. For example, you might use it to rsync the nightly databasedump to another machine for backup purposes.

See Section 19.7 for more information on which data to back up.

The following example uses rsync to move the nightly dump, the Bayes data, and a tarball of/etc/mail/ to a new location:

#!/bin/shrsync /var/spool/Canit-Spam-DB-Backups/SPAM-DATABASE-BACKUP \

/backups/SPAM-DATABASE-BACKUPrsync -av /var/spool/MD-Bayes/DB/ /backups/var/spool/MD-Bayes/DB/tar jcvf /tmp/etc-mail.tar.bz2.tmp /etc/mail/ \

&& mv -f /tmp/etc-mail.tar.bz2.tmp /backups/etc-mail.tar.bz2

Note: In this example /backups/ is a directory on which some external storage is mounted. Replace/backups/ with -essh <user@address>:/location/ for rsync to move the files to aremote server instead.



Note: The /var/spool/MD-Bayes/ directory and its descendants are sensitive to file ownership andpermissions, so preserve them if possible or be sure to reset them after a restore if not. See Section 19.2for more information.

E.4 Hooks

A hook is a script that you supply and that CanIt-Domain-PRO runs when certain events occur. Tocreate a hook, simply create a script in the directory /usr/share/canit/hooks. The script musthave the same name as the hook name (described below) and must be executable and readable.

The following hooks are defined:

• post-cron — runs as rootThis hook is run just before the nightly cron job finishes.

• failed-over — runs as rootThis hook is run on the backup database server just as failover is initiated. Note that the hookruns before failover has actually taken place.

• remove-node-from-cluster — runs as rootThis hook is called when a node is about to be removed from the cluster by the canit-remove-node-from-cluster script.

• reinsert-node-into-cluster — runs as rootIf mail queues do not drain in time when attempting to remove a node from the cluster, CanItreinserts the node into the cluster and calls this hook.

• shut-down-node-post-removal — runs as rootThis hook is called just before a node that has been removed from the cluster is shut down.

• pre-start — runs as rootThis hook is called when /etc/init.d/canit-system start is invoked. If it exits with a non-zerostatus, then CanIt startup is aborted!

• gen-sendmail-maps-failed — runs as rootThis hook is called if the script that generates new Sendmail map files from the Domain Routingtable fails.

• gen-sendmail-maps-succeeded — runs as rootThis hook is called if the script that generates new Sendmail map files from the Domain Routingtable succeeds. It is called after the new mailertable and access files have been generated.

E.5 Migrating CanIt-Domain-PRO to a Different Machine

The following instructions will guide you through migrating to a different server. It is necessary to stopprocessing mail during the migration. The amount of time this will take depends mostly on the size


E.5. MIGRATING CANIT-DOMAIN-PRO TO A DIFFERENT MACHINE 349

of your database. Review the time-stamps on START-BACKUP-TIME and STOP-BACKUP-TIMEin /var/spool/Canit-Spam-DB-Backup for an idea of this time-frame. Assume restoring thedatabase will take 4/3 as long as dumping it does.

By default mail will be tempfailed during the migration. Most sending mail servers will not givewarning Delivery Status Notifications back to the sender unless attempts to deliver to you have failedfor 4 hours.

If you determine that your migration down-time will be too long, there are two options: (1) Allow mailto flow through un-scanned; (2) implement databaseless filtering. Please contact Roaring PenguinTechnical Support for details on these options.

You may wish to upgrade your CanIt-Domain-PRO installation at the same time.

E.5.1 CanIt-Domain-PRO Clusters

If you have a cluster of CanIt-Domain-PRO servers there may be additional considerations. If you aremigrating the database server it will be necessary to stop all CanIt-Domain-PRO servers during themigration since the database will not be available during the dump and restore.

It is safe to install the latest version of CanIt-Domain-PRO on the new machine as long as all membersof the cluster are also upgraded. In a cluster all servers must run the same version of CanIt-Domain-PRO.

The migration procedure includes the necessary steps for clusters. However, you must consult theClustering Guide after migrating to ensure that all machines in the cluster remain properly configured.

E.5.2 Storage Manager

If you are running Storage Manager there may be additional considerations. If you are running onlyone Storage Manager node and it is on the server being migrated then your Storage Manager data mustbe moved during the migration as well. However, in most cases it takes too long to copy this data fromone machine to another.

Therefore, in this case, the recommended procedure is to keep your old CanIt-Domain-PRO serverrunning Storage Manager in read-only mode for some time after the migration is complete. You mayeither: (a) leave the old server running until its Storage Manager data expires (typically 30 days); or(b) begin copying the Storage Manager to the new machine once migration is complete. The latteroption should take much less time (a few days) but requires extra steps.

If you run multiple Storage Manager nodes and have configured your cluster to store at least twocopies of all data, then migration will not be a problem since the other nodes will carry all of the data.

The migration procedure includes the necessary steps for Storage Manager (both options listed above).

E.5.3 Migration Procedure

1. Install CanIt-Domain-PRO on the new server. You may install the latest version.

Note: Please ensure that CanIt-Domain-PRO is fully installed on the new server. In particular, Post-



greSQL roles must be initialized with the canit-prepare-system command, even thoughyou will restore from another database shortly. It is not necessary to run this command on a freshCanIt-Domain-PRO Appliance/ISO install, although it is safe to do so.

2. Stop CanIt-Domain-PRO on the new server, the existing server, and all existing cluster mem-bers:

# /etc/init.d/sendmail stop

# /etc/init.d/canit-system stop-gracefully

(If your version of CanIt-Domain-PRO is old, you might need to use stop instead ofstop-gracefully.)

Disable the CanIt-Domain-PRO web interface for non-admin users by visiting Administra-tion : Disable/Enable.

Note: If your CanIt-Domain-PRO version is older, this function may not be present. If this is the case,or to be extra careful, stop Apache entirely (e.g. /etc/init.d/apache2 stop).

These services must be stopped to ensure that no process attempts to access the database whilethe dump or restore is occurring. To completely ensure safety, you may also set PostgreSQL tolisten only on the loopback address:

Find postgresql.conf and set the listen addresses parameter to ’localhost’.Be sure that only one such parameter exists in the file. Restart PostgreSQL for this to take effect.

Note: After this step CanIt-Domain-PRO is no longer processing mail.

3. Dump the database to a file:

$ pg dump -U postgres spam > spam-dump-file.txt

Note: This command may run for a long time without producing any output. This is normal.

4. Copy the file to the new server. This can be done with ssh:

# scp spam-dump-file.txt root@new machine:/root

5. Copy the entire directory tree rooted at /var/spool/MD-Bayes to the new machine, beingsure to preserve ownership and permissions. There are various ways to do this. However, in thecommon case in which the old and new machine both have rsync and ssh installed:

# rsync --archive -essh /var/spool/MD-Bayes new machine:/var/spool

You may wish to add the --verbose and --progress flags if you have a lot of data tocopy.

6. If your domains were entered manually into /etc/mail/mailertable and/etc/mail/access, you must transfer that information to the new server. If yournew server now has Setup : Domain Routing in its Web Interface you should enter thedomains there once the database is successfully restored.

You may wish to make a tarball of /etc/mail/ to keep as a reference on the new ma-chine. This is handy if you later need to refer to files such as access, mailertable,sendmail.mc, etc.


E.5. MIGRATING CANIT-DOMAIN-PRO TO A DIFFERENT MACHINE 351

Note: WARNING: Do not overwrite /etc/mail/ on the new server!

# tar zcvf old-etc-mail.tar.gz /etc/mail

# scp old-etc-mail.tar.gz root@new machine:/root

7. On the new machine, restore the database from your file:

# dropdb -U postgres spam

# createuser -U postgres -S -D -R spam

(The createuser command may fail if the spam user already exists.)

# createdb -U postgres -E sql-ascii -l C -T template0 spam

# psql -U postgres spam < /root/spam-dump-file.txt

# psql -U postgres spam -c ’ANALYZE VERBOSE’

# canit-prepare-system

Note: psql -U postgres spam < /root/spam-dump-file.txt will produce output.However when it restores the largest table it may appear as though the process has frozen.It is processing a very large table and may take a long time before further output is generated.

8. Log into the Web Interface on the new server and go to Administration : Enable/Disable anddisable CanIt-Domain-PRO if it is not already disabled. This prevents non-admin access to theWeb Interface for recent versions.

This is also a good time to ensure that all your domains are entered into Setup : DomainRouting if your new server has this function.

9. Storage Manager migration 1: skip this step if you do not need to migrate your Storage Managerdata.

Configure the networking on the old server so that it will be accessible when the new server’sfinal networking configuration is complete.

10. Storage Manager migration 2: skip this step if you do not need to migrate your Storage Managerdata.

Access the Web Interface for Storage Manager (see section 16.2.2) and update the networkinginformation. The old server must be set as a Read-Only Hostname. Update the new server’shostname in the Hostnames if it will be different when migration is complete.

11. Clear out all hosts from the cluster members table and system check tables:

# psql -U postgres spam -c ’DELETE FROM cluster members’# psql -U postgres spam -c ’DELETE FROM cluster sanity check’# psql -U postgres spam -c ’DELETE FROM cluster sanity check state’

12. Cluster considerations: skip this step if you do not have a cluster.

Review the Cluster Checklist in the Clustering Guide to ensure that all cluster members areconfigured correctly in your new post-migration configuration.

For example, the scanners may need to have their database IP address updated.



Note: Before proceeding, ensure all cluster members are running the same version.

13. Reconfigure the networking on the new server to its final configuration if necessary.

14. Restart CanIt-Domain-PRO on the new server and all cluster members:

# /etc/init.d/canit-system start

# /etc/init.d/sendmail start

Note: Mail may now flow. Adjust any firewalls, networking equipment or MX records if necessary.

15. Click on Setup : Cluster Management and make sure the expected hosts are all present andcorrectly configured. Adjust any settings as required.

You may need to re-run the commands from step 11 to clear the cluster members and systemcheck tables. After doing so, restart CanIt-Domain-PRO on the new database server beforerestarting other cluster members.

16. Storage Manager migration 3: skip this step if you do not need to migrate your Storage Managerdata, or if you have chosen to allow the old server to remain operating until all data has expiredrather than copying it.

Begin copying the Storage Manager data from the old server to the new:

# rsync --archive --verbose --progress -essh \/var/lib/canit-storage-manager root@newmachine:/var/lib/

Note: This operation may take a very long time, perhaps many hours or even days. You may omit the--verbose or --progress Rsync flags if you don’t want to monitor the Rsync progress.

Go to the Storage Manager Wizard in the Web interface and remove the old server from theRead-Only Hostnames. You may now shut down the old server.

E.6 Cloning a CanIt-Domain-PRO Machine

If you clone a CanIt-Domain-PRO machine, either with disk-imaging software or a virtual envi-ronment’s cloning mechanism, do not bring up the cloned machine without first deleting the file/etc/mail/canit/canit-cluster-member-id. Otherwise, the cluster management sys-tem will assume the cloned machine is still the original machine and will become confused.


Appendix F

Using CanIt-Domain-PRO withmemcached

F.1 Introduction

Memcached is a “distributed memory object caching system.” CanIt-Domain-PRO can use mem-cached to cache the results of Verification Server Lookups. In future, it might make more extensiveuse of memcached to improve performance.

F.2 Using memcached

To use memcached with CanIt-Domain-PRO, you need to install memcached and then configureCanIt-Domain-PRO to use it.

F.2.1 Installing memcached

On our Debian-based CanIt-Domain-PRO appliances, you can install memcached and its client li-braries by running the following command as root:

# apt-get install memcached libcache-memcached-perl php5-memcache

On other platforms, you’ll have to use your system’s package manager to install memcache, theCache::Memcached Perl module, and the Memcache PHP extension.

F.2.2 Configuring memcached

Configuring memcached is beyond the scope of this manual. Consult the memcached documentationfor details on setting the various memcached configuration options.


http://memcached.org/

354 APPENDIX F. USING CANIT-DOMAIN-PRO WITH MEMCACHED

F.2.3 Single vs. Multiple Caches

On a CanIt-Domain-PRO cluster, memcached can be run in one of two basic ways:

1. A single cache. In this case, each cluster member communicates with the same memcacheddaemon (or set of daemons.) There is one cache for the entire cluster.

2. Separate caches. In this case, each cluster member runs its own copy of memcached. Eachmemcached daemon is used only by the node on which it is running.

Each mode has its advantages and disadvantages. A single cache allows more data to be cached andmakes cache hits more likely. However, if a cache node fails, then the entire cluster will be affected.Timeouts when doing cache lookups could negatively affect performance.

If you use separate caches, then less data can be cached and cache misses become more common.However, the failure of one node does not affect any other nodes in the cluster.

To avoid a single point of failure, therefore, we recommend using separate caches: Each CanIt-Domain-PRO cluster member should run its own instance of memcached.

F.2.4 Configuring CanIt-Domain-PRO to use memcached

Once you have installed memcached and configured it to run, edit the CanIt-Domain-PRO configura-tion file (typically /etc/mail/canit/canit.conf) and add a [cache] section. This sectionshould look something like this:

[cache]use_cache = yesdriver = memcachedservers = 127.0.0.1:11211single_cache = nocache_valid_recipients = yescache_invalid_recipients = no

The lines have the following meanings:

• use cache = yes is required to enable caching. Otherwise, CanIt-Domain-PRO will not usememcached.

• driver = memcached is required. In the future, other drivers may be supported, but for now,only memcached is.

• servers = server list specifies the memcached servers. It should be a comma-separated list ofserver descriptions. Each server description is a host name or IP address followed by a colonand the TCP port number on which memcached is listening.

• single cache = no specifies that each CanIt-Domain-PRO cluster member runs (and uses) itsown independent copy of memcached. If the entire cluster uses the exact same set of memcachedservers, then set single cache to yes.


F.3. WHAT IS CACHED 355

• cache valid recipients = yes specifies that CanIt-Domain-PRO should cache valid recipientresults from verification servers. A setting of yes is recommended.

• cache invalid recipients = no specifies that CanIt-Domain-PRO should not cache invalid re-cipient results from verification servers. A setting of no is strongly recommended unless youknow with absolute certainty that the back-end verification server only ever rejects invalid recip-ients and never rejects valid recipients. Some back-end servers may reject valid recipients forpolicy reasons and this could cause CanIt-Domain-PRO to incorrectly cache a valid recipientsas being invalid.

Once you have edited canit.conf, restart CanIt-Domain-PRO.

F.3 What is Cached

Currently, CanIt-Domain-PRO caches verification server results only. It caches a valid recipient for24 hours, and an invalid one for one hour. This can reduce the number of times CanIt-Domain-PROneeds to connect via SMTP to the verification server, and generally improves the recipient-checkingtime by a factor of two or more.


356 APPENDIX F. USING CANIT-DOMAIN-PRO WITH MEMCACHED


Appendix G

Using CanIt-Domain-PRO withPgBouncer

G.1 Introduction

PgBouncer is a “Lightweight connection pooler for PostgreSQL” developed by Skype and available athttp://pgbouncer.projects.postgresql.org/.

PgBouncer is very effective at reducing database load in large CanIt-Domain-PRO installations by re-ducing the number of simultaneous PostgreSQL processes. CanIt-Domain-PRO works very well withPgBouncer in “Transaction Pooling Mode”, which makes very effective reuse of existing PostgreSQLprocesses.

G.2 Installation

Note: We will only describe the installation and operation of PgBouncer with our Debian-based CanIt-Domain-PRO appliance build. Although it is possible to use PgBouncer on other systems, this isnot officially supported; you’ll have to install and configure PgBouncer yourself on those systems.

To install PgBouncer on a CanIt-Domain-PRO appliance, type:

# apt-get update# apt-get install pgbouncer

G.3 Configuration

The PgBouncer configuration files are located in /etc/pgbouncer. The files are:

• userlist.txt: A list of PgBouncer users and how they map to PostgreSQL users.

• pgbouncer.ini: The main PgBouncer configuration file.


http://pgbouncer.projects.postgresql.org/

358 APPENDIX G. USING CANIT-DOMAIN-PRO WITH PGBOUNCER

If you use PgBouncer, you should run one PgBouncer instance on each CanIt-Domain-PRO clustermember.

G.3.1 Configuring userlist.txt

Configuring /etc/pgbouncer/userlist.txt is very easy. It should contain exactly the fol-lowing content:

"spam" "spam""postgres" "postgres"

G.3.2 Configuring pgbouncer.ini

There is a sample pgbouncer.ini file installed in /usr/share/canit. You should copy itinto /etc/pgbouncer and then edit it.

pgbouncer.ini has several sections. They should be configured as follows:

• The [databases] section should point all databases at your database host. If your databasehost is db.example.org, then there should be a single line in the [databases] sectionthat reads:

* = host=db.example.org

That line tells PgBouncer to contact db.example.org for all databases.

Note: If you use a host name for the database host, that name must have an A record in the DNSbecause pgbouncer does not use hostnames defined in /etc/hosts. If your database hostdoes not have a proper DNS entry, use an IP address rather than hostname in pgbouncer.ini.

• The [pgbouncer] section has a wide variety of settings. The defaults in the sample file areprobably fine; if you need to tweak them, see the pgbouncer man page.

The sample configuration file causes pgbouncer to listen for client connections on port 6432.

G.3.3 Configuring CanIt-Domain-PRO to use PgBouncer

Once PgBouncer has been installed and configured, you need to tell CanIt-Domain-PRO to use it. Todo this, edit /etc/mail/canit/canit.conf and change the following settings:

• In the [database] section, set:

db host=127.0.0.1db port=6432

• Also in the [database] section, set:

use pgbouncer=1

This is important if you use failover; it tells the failover code to edit pgbouncer.ini ratherthan canit.conf when changing the database server during failover.


G.3. CONFIGURATION 359

Once PgBouncer has been configured, run /etc/init.d/canit-system restart-gracefullyon all cluster nodes.


360 APPENDIX G. USING CANIT-DOMAIN-PRO WITH PGBOUNCER


Appendix H

CanIt-Domain-PRO Logging

H.1 General Information

CanIt-Domain-PRO logs messages regarding its operation using syslog. By default, these are loggedusing the mail syslog facility to keep them together with Sendmail’s logs. This is recommended,but if for some reason you wish to change it, you can do so by modifying the syslog facilityconfiguration setting in the mimedefang section of /etc/mail/canit/canit.conf.

In general, a CanIt-Domain-PRO log entry will consist (after the standard syslog preamble of date,host, process name, and process ID) of the word CanIt: followed by the 14 character Sendmailqueue ID (or the text NOQUEUE) followed by another colon. After this comes the message-specificinformation for that log type.

Several types of log message are generated, at different log levels:

Debugging messages Debugging messages provide very verbose, detailed information regarding theinternal workings of CanIt-Domain-PRO. These are logged using syslog’s debug facility, andare turned off by default in shipped versions of CanIt-Domain-PRO.

You will probably never need to enable debug logging, but if you need to do so, you must editthe CanIt-Domain-PRO filter file (/etc/mail/canit/canit-domain-pro-filter)and add the line:

CanIt::Logger::set debuglevel( CanIt::Logger::DEBUG ON() );

to the filter initialize() function, and restart the CanIt-Domain-PRO service.

When enabled, debug logging provides extra debugging information. After the general logentry info mentioned above, a debug message consists of DEBUG:, the message itself, and thenin parentheses, the line, file, function, and caller information for each debug message.

Note: Enabling debug logging is not recommended on a heavily loaded production server, as the extrasyslog traffic will slow things down, and greatly increase the disk space required for your logs.

Regular log messages Regular log messages provide information about the normal operation ofCanIt-Domain-PRO and are logged at the ’info’ level.


362 APPENDIX H. CANIT-DOMAIN-PRO LOGGING

Event messages Event log messages provide information about the normal operation of CanIt-Domain-PRO in a format that is both human readable and machine parseable. These are loggedat the ’info’ level.

Warning messages Warning messages indicate that an undesirable, but non-fatal, condition has oc-curred. These are logged at the ’warning’ level.

Error messages Error messages indicate that a failure has occurred within CanIt-Domain-PRO andshould be attended to immediately. These are logged at the ’error’ level.

H.2 Event Log Format

Event messages are logged in a format designed to be both human-readable and machine-parseable.This format consists of comma-separated key=value pairs, where the key consists of entirely lower-case alphabetic characters, and the value consists of arbitrary text appropriate for that key, with prob-lematic characters such as newlines and commas replaced with a % followed by their two-digit hex-adecimal value.

With the exception of what, which always appears first, and subject, which will appear last ifpresent, the key/value pairs cannot be assumed to occupy any specific position in the log line. De-pending on where and why the message was logged, different keys will be present.

An example log message is:Jan 01 13:10:31 oxygen mimedefang.pl[9813]: CanIt: j4CHAVtu009864:what=accepted, nrcpts=1, relay=192.168.10.8, score=2.5,[email protected], stream=user1,tests=HTML MESSAGE, subject=Yes%2C this is an example

(We have wrapped the output for readability; in reality, the log message would appear on a single line.)

Here we see the standard date, time, hostname, process name, and process ID from syslog, the nameCanIt:, the sendmail queue ID for the message being processed, and a number of key-value pairsseparated by commas.

The keys that can appear in an “event” log line are:

what This field provides the first indication of what happened to the message. The ’reason’ and’detail’ fields provide further information

Valid values for ’what’ are:

accepted Message was accepted and relayed through. The ’reason’ field may contain oneof: approved, sender-whitelisted, domain-whitelisted, host-whitelisted, unscanned-toobig,skip-spam-scan, opt-out, or no reason at all if none of those cases apply.

rejected Message (or sender, or recipient) was rejected and the sending relay was givena 5xx failure code. The ’reason’ field may contain one of: auto-reject, auto-reject-no-incident, blacklisted-recipient, domain-blacklisted, exe, ext, host-blacklisted, invalid-recipient, mime, rbl-blacklisted, sender-blacklisted, too-large, or virus.

tagged Message was tagged and relayed through. what=tagged log lines will not contain a’reason’ field.


H.2. EVENT LOG FORMAT 363

discarded Message was discarded silently. The ’reason’ field can be auto-reject, auto-reject-no-incident, exe, ext, mime, virus.

pending Message was quarantined and held for human review.

greylisted Message was greylisted with a 4xx code. what=greylisted lines will not contain areason field.

reason This provides secondary information (the ”why” to the ”what” above) regarding the disposi-tion of an incoming connection. Valid values are:

approved Message was manually approved from the quarantine interface.

auto-reject Message was rejected. An incident is available and is indicated by the value for theincident key.

auto-reject-no-incident Message was automatically rejected due to spam score, and no inci-dent was created.

blacklisted-recipient The specified recipient was blocked

domain-blacklisted The domain of the sender’s address was blocked in the specified stream.

domain-whitelisted The domain of the sender’s address was always allowed in the specifiedstream.

exe The message contained a file with an extension considered executable on Microsoft oper-ating systems. detail will contain the extension name. Note that CanIt-Domain-PROno longer generates the exe reason, but older versions used to. New versions of CanIt-Domain-PRO only generate the ext reason.

ext The message contained a file with a blocked extension. detail will contain the extensionname.

host-blacklisted The relay host was blocked in the specified stream.

host-whitelisted The relay host was always allowed in the specified stream.

invalid-recipient The specified recipient was not valid.

mime The message contained a file with a blocked MIME type. detail will contain theactual MIME type found.

opt-out The stream containing this message is configured to opt out of spam scanning.

rbl-blacklisted The relay sending this message was blocked by an RBL entry.

sender-blacklisted The sender address was blocked in the specified stream.

sender-whitelisted The sender address was always allowed in the specified stream.

skip-spam-scan The originating relay was in a Known Network marked with “Skip SpamScan”

too-large The message was rejected because it was over the configured maximum size formessages received. The detail key will contain the actual size of the message.

unscanned-toobig The message was not scanned for spam because it was over the configuredmaximum size for scanning and could not be reduced below that size. The detail keywill contain the actual size of the message.



virus The message contained a virus payload. The detail key will contain the name of thevirus found.

attach types This provides details about attachment types found in the message. It consists of asemicolon-separate list of filename extensions. Any filenames found inside an archive file areprefixed with >.

For example, if an email message contains a PNG image attachment and a ZIP file, and the ZIPfile contains a DLL file, CanIt-Domain-PRO may log something like this:

attach types=png;zip;>dll

detail This provides further detail if necessary (and available) from certain tests. For example, ifwhat=discard and reason=virus, the detail key will contain the name of the virus found.

city The name of the city in which the SMTP sending relay is located, if it could be determined.

country code The two-letter ISO-3166 country-code in which the SMTP sending relay is located, ifit could be determined.

incident The numeric ID of the incident, if available. An incident ID will be available only if anincident is associated with this message, either because it was created, or because the messagematched an existing incident.

resolved by If an incident was present for this message, this field provides the username of the userresponsible for accepting or rejecting the message.

nrcpts The number of recipients for the given message. In general, rather than listing the individualrecipients (which, in some cases could number in the hundreds), we use this key to provide onlythe number. The exception is when a particular single recipient is affected. In that case, we usethe recipient key to log the actual address.

recipient If an envelope recipient is rejected for some reason, the recipient address is logged with thiskey.

relay The IP address of the sending relay. If parsing of Received: headers is enabled, this containsthe address retrieved from the headers. Otherwise, the actual connecting relay IP is logged.

score The score for the message, if scoring rules were applied.

sender The envelope sender of the message.

header from The From: header address of the message. This is only logged if it is different from thesender key. It is also not logged if the message content is not yet available (for example, if anonexistent recipient is rejected.)

subject The subject line of the message, if available. This key always appears last in the log message.

stream The name of the stream being applied to the message at the time.

tests A semicolon-separated list of test names (both SpamAssassin and CanIt tests) that triggered forthe message.


H.2. EVENT LOG FORMAT 365

os The SMTP client’s operating system name as determined by passive OS fingerprinting. Typicallysomething like “Windows” or “Linux”.

osver The SMTP client’s operating system version as determined by passive OS fingerprinting.

linktype The SMTP client’s Internet link type as determined by passive OS fingerprinting.

realm The name of the realm in which the message was processed.


Appendix I

SNMP Agents for CanIt-Domain-PRO

I.1 Introduction

SNMP (“Simple Network Management Protocol”) is a protocol for monitoring networks. An SNMPmonitoring station typically polls an SNMP Agent via UDP and receives data about the monitoredfacility.

CanIt-Domain-PRO includes an SNMP agent that integrates with the Net-SNMP package. (For moreinformation on Net-SNMP, please see http://www.net-snmp.org/)

Note: This chapter is not a tutorial on SNMP, nor will it tell you how to configure Net-SNMP; we assumeyou’re familiar with both. Also, we support the SNMP agents only on our Debian-based applianceand our Red Hat Enterprse Linux RPMs; on all other systems, the SNMP agents are supplied on anas-is basis without support.

To use the SNMP agent, ensure that Net-SNMP is installed, and that snmpd is configured and set tostart on system boot.

Data returned by an SNMP agent is described in a Management Information Base or MIB. You candownload the CanIt-Domain-PRO MIB by logging in to the Web interface as the administrator, select-ing Setup : Wizards and then clicking Download CanIt SNMP MIB File.

I.2 The SNMP Agent

The SNMP agent included with CanIt-Domain-PRO monitors:

1. Sendmail queue sizes and number of processes.

2. MIMEDefang busy/free scanners.

3. Failover status.

The agent provides information under the MIB tree .1.3.6.1.4.1.10055, which corresponds toenterprises.roaringpenguin.


http://www.net-snmp.org/

368 APPENDIX I. SNMP AGENTS FOR CANIT-DOMAIN-PRO

I.2.1 Enabling the agent

To enable the SNMP monitoring agent, add this line to snmpd.conf:

pass_persist .1.3.6.1.4.1.10055 /usr/share/canit/scripts/canit-snmp-agent

Additionally, the cron job /usr/share/canit/scripts/canit-snmp-cron must be setup to run once per minute. On CanIt-Domain-PRO appliances, edit the file /etc/cron.d/canit-snmp and uncomment the line. On other platforms, create a cron script that runs/usr/share/canit/scripts/canit-snmp-cron as root once per minute.

I.2.2 Configuring SNMPd

You may need to configure your SNMP daemon to allow connections from your external monitoringservices. The following instructions apply to the Appliance Build. For other operating systems,consult the distributor’s documentation or support resources.

To configure the SNMP daemon to listen on your external network interface, edit/etc/snmp/snmpd.conf. Find this line:

#agentAddress udp:161,udp6:[::1]:161

and uncomment it by deleting the leading # sign.

To tell the SNMP daemon to allow readonly connections, edit /etc/snmp/snmpd.conf. Add aline like this:

rocommunity public default

You may prefer a different COMMUNITY name than public. If so, simply change the publicparameter to something else. You can also restrict access to the SNMP daemon by using a networkand mask such as 10.0.0.0/16 in place of default in the above line.

Finally, restart the daemon and test with the following two commands:

/etc/init.d/snmpd restartsnmpwalk -v 1 -c public localhost .1.3.6.1.4.1.10055

Note: If the snmpwalk command doesn’t give any output, wait a minute before trying again. The daemonmay take a moment to fully start up, or the cron job may not have run yet.

I.2.3 Agent Data

The Sendmail portion of the agent returns information about the number of entries in the main queueand the submission queue. The meanings of the variables are as follows; “sendmail” is short for.1.3.6.1.4.1.10055.100:


I.2. THE SNMP AGENT 369

• sendmail.1.1.1.1 — constant integer 1

• sendmail.1.1.2.1 — constant string “Main Queue”

• sendmail.1.1.3.1 — number of messages in Sendmail’s primary queue


• sendmail.1.1.2.2 — constant string “Submission Queue”

• sendmail.1.1.3.2 — number of messages in Sendmail’s submission queue


• sendmail.1.1.2.3 — constant string “Sendmail Process Count”

• sendmail.1.1.3.3 — number of sendmail processes running

The MIMEDefang portion of the agent returns information about the number of messages processedin the last 10 seconds, 1 minute, 5 minutes and 10 minutes; the average scan time in milliseconds, andthe average number of busy scanners. The meanings of the variables are as follows; “mimedefang” isshort for .1.3.6.1.4.1.10055.1:

• mimedefang.1.1.1.1 — constant integer 1



• mimedefang.1.1.2.1 — constant string “Max workers”

• mimedefang.1.1.2.2 — constant string “Busy workers”

• mimedefang.1.1.2.3 — constant string “Free workers”

• mimedefang.1.1.3.1 — maximum number of scanning processes configured

• mimedefang.1.1.3.2 — number of busy scanning processes

• mimedefang.1.1.3.3 — number of free scanning processes





• mimedefang.2.1.2.1 — constant string “10 Seconds”

• mimedefang.2.1.2.2 — constant string “1 Minute”

• mimedefang.2.1.2.3 — constant string “5 Minutes”


370 APPENDIX I. SNMP AGENTS FOR CANIT-DOMAIN-PRO

• mimedefang.2.1.2.4 — constant string “10 Minutes”

• mimedefang.2.1.3.1 — number of messages scanned in the last 10 seconds

• mimedefang.2.1.3.2 — number of messages scanned in the last 1 minute

• mimedefang.2.1.3.3 — number of messages scanned in the last 5 minutes

• mimedefang.2.1.3.4 — number of messages scanned in the last 10 minutes

• mimedefang.2.1.4.1 — average scan time in milliseconds times 1000 (last 10 seconds)

• mimedefang.2.1.4.2 — average scan time in milliseconds times 1000 (last 1 minute)

• mimedefang.2.1.4.3 — average scan time in milliseconds times 1000 (last 5 minutes)

• mimedefang.2.1.4.4 — average scan time in milliseconds times 1000 (last 10 minutes)

• mimedefang.2.1.5.1 — average busy scanners times 1000 (last 10 seconds)

• mimedefang.2.1.5.2 — average busy scanners times 1000 (last 1 minute)

• mimedefang.2.1.5.3 — average busy scanners times 1000 (last 5 minutes)

• mimedefang.2.1.5.4 — average busy scanners times 1000 (last 10 minutes)

Note: The average scan time and average busy scanners values are reporting an average, which is normally afloating-point value internally. Since SNMP does not have a floating-point type, we multiply the rawvalue by 1000 so that the information can be reported using a SNMP integer type with an acceptablelevel of precision.

If you have configured PostgreSQL failover as described in the CanIt-Domain-PRO Clustering Guide,you will also be able to retrieve information about the status of PostgreSQL failover. The meanings ofthe variables are as follows; “failover” is short for .1.3.6.1.4.1.10055.2:

• failover.1 — the type of server; one of “master” or “backup”.

• failover.2 — if 1, then the failover system is OK. If 0, then there are problems that should beinvestigated.

• failover.3 — a count of errors seen in the PostgreSQL log file. (Meaningful only on the “master”server.)

• failover.4 — a count of WAL files waiting to be consumed on the backup server. Always re-ported as 0 on the master server.

• failover.5 — the count of WAL files in the pg xlog directory on the master server. Alwaysreported as 0 on the backup server.

• failover.6 — the time of last base backup as a UNIX timestamp (seconds since 1 January 197000:00:00 UTC.) Always reported as 0 on the master server.

• failover.7 — the time the last WAL file was shipped to the backup server as a UNIX timestamp.Always reported as 0 on the master server.


Appendix J

Additional Scripts

CanIt-Domain-PRO ships with additional scripts that you may find useful. Please note that thesescripts are not officially supported by Roaring Penguin Software Inc.

J.1 reset-password.pl

The script /usr/share/canit/scripts/reset-password.pl lets you reset the adminis-trator password if you forget it. To run the script, simply type:

# /usr/share/canit/scripts/reset-password.pl

and follow the prompts.


372 APPENDIX J. ADDITIONAL SCRIPTS


Appendix K

Bayes Database Back-Ends


K.1 PostgreSQL Bayes Data Storage

By default, versions of CanIt-Domain-PRO prior to 3.2.0 store Bayesian statistics in the PostgreSQLdatabase in a table called bayes. At a large site, Bayesian lookups can cause considerable databasetraffic and substantial load on the database machine. CanIt-Domain-PRO has a mechanism to storeBayesian statistics in CDB database files. These files are local to each scanner. Lookups are extremelyfast, and involve no database traffic and no load on the PostgreSQL database. Similarly, updates donot involve the PostgreSQL database, which can greatly improve performance.

Note: As of CanIt-Domain-PRO version 3.3.0, the PostgreSQL back-end is no longer supported, and cannotbe used.

K.2 Berkeley Database Bayes Storage

Versions 6.0.x and earlier of CanIt-Domain-PRO used BerkeleyDB to store Bayesian statistics. Incurrent versions, we now use CDB for the same data. CanIt-Domain-PRO can now read both Berke-leyDB and CDB Bayesian statistics files, but will only write CDB files. As such, no conversion stepis necessary – all Bayesian statistics will be migrated to CDB storage as new training is performed.

K.3 CDB Database Bayes Storage

CDB storage of Bayes data operates as follows:

• The master database files are stored on the machine running the ticker. Each stream has its owndatabase file under the directory /var/spool/MD-Bayes/DB.


374 APPENDIX K. BAYES DATABASE BACK-ENDS

• Bayes training is performed by the ticker. It updates the master CDB database files. If you arerunning a cluster, the ticker then copies the updated database files to each scanning machine.

As a consequence of the way the CDB database files work, you must be aware of the following:

• You must have sufficient room under /var/spool/MD-Bayes/DB for all of your Bayesdata on the ticker machine and on each scanner.

• If you want to back up your Bayes data, you must back up /var/spool/MD-Bayes on theticker machine as well as backing up the nightly database dump.

• The ticker machine must be able to communicate via SSH to all scanning servers. SSH keysetup is performed automatically, so no additional configuration should be necessary beyondthe setup of a proper CanIt-Domain-PRO cluster (see Section K.4.)

K.4 Cluster Considerations

CDB files need to copy the files to all your scanning machines. On a new cluster, this will be takencare of with update propagation (see below). However, if you are adding a new server to an existingcluster, you will need to copy over your data. If you have rsync and ssh installed, the followingcommands can be used to copy the data over. They should be run as defang on the ticker machine;we assume $SCANNERS is a list of all your newly-added scanners.

for mach in $SCANNERS ; dorsync -essh --archive --progress --verbose /var/spool/MD-Bayes/DB \

$mach:/var/spool/MD-Bayesdone

K.4.1 Propagating Updates

Because the ticker can only update CDB databases locally on the ticker machine, a mechanism is re-quired to copy updated files to all scanning machines. In recent versions (post-6.0.3) of CanIt-Domain-PRO, this is performed via the standard cluster communication process using an automatically-generated shared SSH key.

Previous versions used sync-berkeley-db and sync-berkeley-db-multi scripts to syn-chronize the data; these are no longer necessary or supported.

K.5 Switching back to PostgreSQL Bayes Storage

As of CanIt-Domain-PRO 3.3.0, it is not possible to switch back to the PostgreSQL storage modulefor Bayes data.


Appendix L

System Check Tests

CanIt-Domain-PRO features an extensive self-test system that checks for common misconfigurationsand emails the administrator if problems are detected. You can see an overview of the self tests on theSetup : System Check page.

The tests are as follows:

ApplianceDebianRepositories If this test fails, then your appliance has non-Roaring Penguin repos-itories in its sources list. We do not recommend this.

ApplianceDebianVersion If this test fails, then your CanIt-Domain-PRO appliance cannot determineits Debian version. Contact Roaring Penguin support personnel for assistance.

ApplianceDiskSpace If this test fails, then at least one filesystem on the CanIt-Domain-PRO appli-ance has less than 10% free disk space.

BaseURLConfigured If this test fails, you have not configured the Base URL of the CanIt-Domain-PRO installation. Run through the Basic Setup Wizard (under Setup : Wizards) to correctthis.

BayesDatabaseFormat If this test fails, the Bayes storage mechanism from an old CanIt-Domain-PRO installation is incorrect. Contact Roaring Penguin support for help.

ClamAVCurrent If this test fails, your ClamAV signatures are out of date. Check the Clam logs tosee what might be causing the problem.

ClusterMain databaseHost If this test fails, no cluster member is designated as the main databasehost. Contact Roaring Penguin support for help.

ClusterScannerHost If this test fails, no cluster member is configured as a scanner. Fix this underSetup : Cluster Management.

ClusterStorageManagerHost If this test fails, the Storage Manager is misconfigured. Run thoughthe Storage Manager Wizard to correct the problem.

ClusterTickerHost If this test fails, no cluster member is configured as a ticker. Fix this underSetup : Cluster Management.


376 APPENDIX L. SYSTEM CHECK TESTS

ClusterWebserverHost If this test fails, no cluster member is configured as a Web server. Fix thisunder Setup : Cluster Management.

CopyToCluster If this test fails, it indicates a problem copying Bayes data from the ticker host toanother host in the cluster. Make sure all hosts can communicate with each other over SSH onTCP port 22.

Cron If this test fails, then the nightly cron job has not run recently. You should immediately investi-gate and take corrective action.

DatabaseDump If this test fails, it indicates that the nightly database dump has failed. You shouldimmediately take corrective action; check the log file canit-cron.log in the directory/var/spool/Canit-Spam-DB-Backup to see what went wrong.

DatabaseVacuum If this test fails, it indicates that the nightly database vacuum has failed. Youshould immediately take corrective action; check the log file canit-cron.log in the direc-tory /var/spool/Canit-Spam-DB-Backup to see what went wrong.

DebianAutoUpgrade (Appliance Only) If this test fails, it indicates that you have set the upgradetype to Automatic, but have a version of PostgreSQL that is too old to safely support automaticupgrades. Until you can upgrade PostgreSQL, you may silence the warning by going to Setup: Wizards and running the Upgrade Configuration Wizard. Set the upgrade type to Manual.

DeprecatedFiles If this test fails, it indicates there are some obsolete files from an old CanIt-Domain-PRO installation. Contact Roaring Penguin support for help.

DeleteOnCluster If this test fails, it indicates a problem deleting Bayes data on a host in the cluster.Make sure all hosts can communicate with each other over SSH on TCP port 22.

FailoverWALCount If this test fails on the backup database server, then you have set up PostgreSQLfailover but the backup PostgreSQL server is not consuming WAL files correctly. If this test failson the master database server, then the PostgreSQL pg xlog directory is filling up with WALfiles. There is likely a problem shipping the WAL files to the backup server.

StreamingReplication If this test fails on the master database server, it means you have configureda hot-standby replica but it is not streaming transactions from the master database. Check thehot-standby database server immediately for problems.

Hostname If this test fails, then your host is called localhost. You should give it a real host name.

HostnameNotLoopback If this test fails, then a host’s canonical name resolves to the loopback ad-dress (i.e, 127.0.0.1 or ::1). This usually indicates a bad entry in /etc/hosts on the affectedmachine. You should ensure that /etc/hosts does not contain an entry for the host pointingit at 127.0.0.1 or ::1. Only localhost should point to the loopback address.

LicenseValid If this test fails, your license key is invalid. Contact Roaring Penguin support for help.

MainDatabase If this test fails, there isn’t exactly one machine in the cluster marked as the maindatabase server. Contact Roaring Penguin support for help.


377

MaxFSMPages If this test fails, PostgreSQL’s max fsm pages parameter is too low. You shouldtake immediate corrective action: Edit the PostgreSQL postgresql.conf file to increasemax fsm pages and restart PostgreSQL.

Note: You may need to increase your kernel’s max. shared memory limit. If PostgreSQL failsto restart after updating max fsm pages, revert your change and restart. Correct thekernel.shmmax issue, then retry.

OldMSAQueueFile If this test fails, there is an old queue file in the Sendmail submission queue.Make sure that Sendmail is running and that a submission queue runner is active.

OldMTAQueueFile If this test fails, there is an old queue file in the Sendmail main queue. Makesure that Sendmail is running and that a main queue runner is active.

PhishListDownload If this test fails, the phishing list download has failed. Check the reason in thedescription field to diagnose why the download failed.

PhishingLinksDownload If this test fails, the phishing URL download has failed. Check the reasonin the description field to diagnose why the download failed.

PostgresEncoding If this test fails, the your spam database uses the wrong encoding. Contact Roar-ing Penguin support for help.

PostgresVersion If this test fails, your version of PostgreSQL is too old. Contact Roaring Penguinsupport for help.

PostgresXXX If this test fails, the corresponding PostgreSQL configuration value is too low. Increaseit in postgresql.conf and restart PostgreSQL.

RecipientVerification:XXX If this test fails, then there is no mechanism to verify recipients for thegiven domain. You should enable recipient verification by doing one of the following:

1. Set up a Verification Server (Section 5.4.)

2. Set up a User Lookup method that validates recipients (Section 7.2.)

3. Use the Valid Recipients Table (see the Users’ Guide.)

RPTNBayesDownload If this test fails, an RPTN download has failed. Check the reason in thedescription field to diagnose why the download failed.

RPTNEnabled If this test fails, you have not enabled RPTN downloads. Run through the RPTNSetup Wizard (under Setup : Wizards) to correct this.

RPTNGeoDownload If this test fails, the geolocation data download has failed. Check the reason inthe description field to diagnose why the download failed.

RPTNRulesDownload If this test fails, a ruleset download has failed. Check the reason in the de-scription field to diagnose why the download failed.

RPTNSynchronization If this test fails, then at least one scanner has no RPTN data (or outdatedRPTN data). Make sure your RPTN downloads are succeeding and that all cluster members cancommunicate via SSH over TCP port 22.



StorageManagerConfig If this test fails, you have enabled the Storage Manager, but have not con-figured any read/write Storage Manager nodes. Run through the Storage Manager Wizard tocorrect the misconfiguration.

SupportValid If this test fails, your support term is about to expire or has expired. Contact RoaringPenguin’s sales department to extend your support.

TickerTable If this test fails, no ticker tasks are running. Contact Roaring Penguin support for help.

TickerTaskXXX If this test fails, the corresponding ticker task has not run recently. Contact RoaringPenguin support for help.

VirusScannerEnabled If this test fails, then no virus scanners are enabled. You should enable a virusscanner in /etc/mail/canit/virus-scanners.pl.

WebserverDeprecatedFiles If this test fails, there are obsolete files in the CanIt-Domain-PRO webdirectory. Contact Roaring Penguin support for help.

L.1 Disabling System Checks

Although we do not recommend disabling system checks, you can selectively disable checks by edit-ing /etc/mail/canit/canit.conf and creating a [sanitychecker] section. Within thatsection, add lines of the form testname=ignore to ignore specific tests.

For example, if you wish to ignore the ApplianceDiskSpace test, add these lines tocanit.conf:

[sanitychecker]ApplianceDiskSpace=ignore

L.2 Anomaly Detection

CanIt-Domain-PRO can detect and report certain anomalies that occur during operation. In additionto being reported to the overall CanIt-Domain-PRO administrators, anomalies are reported to realmadministrators. A given realm administrator can see all anomalies for his or her realm, subrealms, etc.

If any anomalies have occurred, you will see the following notice when you first log in to CanIt-Domain-PRO:

Figure L.1: Anomaly Notice

To see details, click on Administration and then Anomalies.

Each anomaly reported consists of the following parts:


L.2. ANOMALY DETECTION 379

• Realm – the realm in which the anomaly occurred.

• Family – the general class of the anomaly.

• Detail – more detail about the anomaly (for example, the server or domain involved, etc.)

• Message – a human-readable error message.

• Time Frame – the time frame over which the anomaly has occurred. This shows the first andlast time the anomaly was observed.

The meanings of the various families are:

• AccountInfo::LDAP – something went wrong with an LDAP lookup. The “Detail” field willcontain the name of the particular User Lookup that failed and the “Message” field will explainwhat happened.

• DesynchronizedDNS – the domain’s name servers disagree about its MX records. You shouldfix the name servers so that all of them report the same set of MX records.

• DKIM – a domain has DKIM signing set up, but there is a problem with the domain’sdomainkey DNS record.

• RecipientVerification – a domain does not correctly validate recipients. The “Detail” field willcontain the domain name. You should enable recipient verification with one of the followingmethods:

1. Set up a Verification Server (Section 5.4.)

2. Set up a User Lookup method that validates recipients (Section 7.2.)

3. Use the Valid Recipients Table (see the Users’ Guide.)

• VerificationServer – something went wrong trying to contact a verification server. The “Detail”field will contain the server list.

L.2.1 Disabling Recipient Verification Anomaly Testing

If you wish CanIt-Domain-PRO to permit a domain not to validate recipients, you can disable theRecipientVerification anomaly test as follows:

• Under Setup : Verification Servers, create a Verification Server entry for the domain wherethe server name is the literal text ignore.

Note: Although it is possible to disable Recipient Verification testing for a domain, we do not recommendthis. Allowing wildcard recipients could result in a large amount of useless scanning and wasted CPUtime.



L.2.2 More Details about Anomalies

If you require additional details about an anomaly, click on the link in the Time Frame column. TheAnomaly Details screen will appear:

Figure L.2: Anomaly Details

The anomaly details page will contain one row for each time the anomaly was logged. The Queue IDcolumn contains the Sendmail queue ID (if any) associated with the message causing the anomaly; ifyou have the Log Searching and Indexing component installed, then the Queue ID will be a link thattakes you directly to the mail logs associated with a given occurrence of the anomaly.

L.2.3 Suppressing Anomaly Notification Emails

Normally, CanIt-Domain-PRO sends out an email to realm administrators once a night if it noticesanomalies. If you wish to suppress these messages, go to Administration : Anomalies and set“Should the system send anomaly notification emails to realm administrator(s)?” to No. This settingis inherited by sub-realms, so if you want to turn off anomaly notifications for a realm but leave themon for a subrealm, you need to switch into the subrealm and override the parent realm setting.


Appendix M

The CanIt-Domain-PRO License

READ THIS LICENSE CAREFULLY. IT SPECIFIES THE TERMS AND CONDITIONS UNDERWHICH YOU CAN USE CANIT-DOMAIN-PRO

This license may be revised from time to time; any given release of CanIt-Domain-PRO is licensedunder the license version which accompanied that release.

CanIt-Domain-PRO is distributed in source code form, but it is not Free Software or Open-SourceSoftware. Some CanIt-Domain-PRO components are Free Software or Open-Source, and we detailthem below:

The following files may be redistributed according to the licenses listed here. An asterisk (*) in a filename signifies a version number; the actual file will have a number in place of the asterisk.

File Licensesrc/Archive-Tar-*.tar Perl Licensesrc/Config-Tiny-*.tar Perl Licensesrc/DBD-Pg-*.tar Perl Licensesrc/DBI-*.tar Perl Licensesrc/Data-ResultSet-*.tar Perl Licensesrc/Data-UUID-*.tar Perl Licensesrc/Digest-MD5-*.tar Perl Licensesrc/Digest-SHA1-*.tar Perl Licensesrc/File-Spec-*.tar Perl Licensesrc/File-Temp-*.tar Perl Licensesrc/HTML-Parser-*.tar Perl Licensesrc/HTML-Tagset-*.tar Perl Licensesrc/IO-Zlib-*.tar Perl Licensesrc/IO-stringy-*.tar Perl Licensesrc/Log-Syslog-Abstract-*.tar Perl Licensesrc/MIME-Base64-*.tar Perl Licensesrc/MIME-tools-*.tar Perl Licensesrc/Mail-SPF-Query-*.tar Perl Licensesrc/Mail-SpamAssassin-*.tar Apache License, Version 2.0src/MailTools-*.tar Perl License


https://www.roaringpenguin.com/licenses/artistic-license.txt


















https://www.roaringpenguin.com/licenses/apache-license-2.0.txt


382 APPENDIX M. THE CANIT-DOMAIN-PRO LICENSE

File Licensesrc/Module-Pluggable-Tiny-*.tar Perl Licensesrc/Net-CIDR-Lite-*.tar Perl Licensesrc/Net-DNS-*.tar Perl Licensesrc/Net-IP-*.tar Perl Licensesrc/Time-HiRes-*.tar Perl Licensesrc/TimeDate-*.tar Perl Licensesrc/URI-*.tar Perl Licensesrc/YAML-Syck-*.tar Perl Licensesrc/clamav-*.tar GPLv2src/p0f-*.tar GPLv2src/libwww-perl-*.tar Perl Licensesrc/mimedefang-*.tar GPLv2

ALL REMAINING FILES IN THIS ARCHIVE (referred to as ”CanIt-Domain-PRO”) ARE DIS-TRIBUTED UNDER THE TERMS OF THE CANIT LICENSE, WHICH FOLLOWS:

THE CANIT LICENSE

1. CanIt-Domain-PRO is the property of Roaring Penguin Software Inc. (”Roaring Penguin”).This license gives you the right to use CanIt-Domain-PRO, but does not transfer ownership ofthe intellectual property to you.

2. CanIt-Domain-PRO is licensed with a limit on the number of allowable protected domains ormailboxes. This limit is called ”the Usage Limit”.

CanIt-Domain-PRO usage may be purchased on a yearly basis, or you may purchase a perpetuallicense.

3. You may use CanIt-Domain-PRO up to the Usage Limit you have purchased. If you havepurchased yearly usage, you may continue to use CanIt-Domain-PRO until your purchasedusage time expires, unless you purchase additional time. If you have purchased a perpetuallicense, you may continue to use CanIt-Domain-PRO indefinitely, providing you do not violatethis license.

If you have purchased yearly usage, you may exceed your purchased limit by up to 10% untilthe yearly renewal date, at which time you must purchase a sufficient limit for the increasednumber of domains or mailboxes.

If you have purchased a perpetual license, or wish to increase your usage more than 10% aboveyour paid-up limit, you must purchase the additional usage within 60 days of the increase.

4. You may examine the CanIt-Domain-PRO source code for education purposes and to conductsecurity audits. You may hire third-parties to audit the code providing you first obtain permis-sion from Roaring Penguin. Such permission will generally be granted providing the third-partysigns a non-disclosure agreement with Roaring Penguin.

5. You may modify the CanIt-Domain-PRO source code for your own internal use, subject to therestrictions in Paragraph 9 below. However, if you do so, you agree that Roaring Penguin is










https://www.roaringpenguin.com/licenses/gpl.txt




383

released from any obligation to provide technical support for the modified software. If youwish your modifications to be incorporated into the mainstream CanIt-Domain-PRO release,you agree to transfer ownership of your changes to Roaring Penguin.

6. You may make backups of CanIt-Domain-PRO as required for the prudent operation of yourenterprise.

7. You may not redistribute CanIt-Domain-PRO in source or object form, nor may you redistributemodified copies of CanIt-Domain-PRO or products derived from CanIt-Domain-PRO.

8. If you violate this license, your right to use CanIt-Domain-PRO terminates immediately, andyou agree to remove CanIt-Domain-PRO from all of your servers.

9. Restrictions on modification:

(a) Notwithstanding Paragraph 5, you may not make changes to CanIt-Domain-PRO or yoursoftware environment which would allow CanIt-Domain-PRO to run without a valid Li-cense Key as issued by Roaring Penguin. You also agree not to set back the time on yourserver to artificially extend the validity of a License Key, or do anything else which wouldartificially extend the validity of a License Key.

(b) You may modify the Web-based interface only providing you adhere to the followingrestrictions:

(c) At the bottom of every CanIt-Domain-PRO web page, the following text shall appear, in asize, color and font which are clearly legible:Powered by CanIt-Domain-PRO (Version x.y.z) from Roaring Penguin Software Inc.where x.y.z is the product version. In addition, “CanIt-Domain-PRO” shallbe a clearly-marked hypertext link to https://www.roaringpenguin.com/powered-by-canit.php

(d) You may not include elements on the CanIt-Domain-PRO Web interface that require plug-ins (such as, but not limited to, Macromedia Flash, RealPlayer, etc.) to function.

(e) You may not include Java applets on the CanIt-Domain-PRO Web interface.

(f) If you include JavaScript on the Web interface, you shall ensure that the interface functionssubstantially unimpaired in a browser with JavaScript disabled.

(g) You shall not include browser-specific elements on the Web interface. You shall ensurethat the Web interface functions substantially unimpaired on the latest versions of thefollowing browsers:

• Internet Explorer for Windows• Mozilla for Windows• Mozilla for Linux• Konqueror for Linux

(h) You may not include banner ads on the CanIt-Domain-PRO Web interface.

10. Disclaimer of Warranty (Virus-Scanning)


https://www.roaringpenguin.com/powered-by-canit.php

https://www.roaringpenguin.com/powered-by-canit.php

384 APPENDIX M. THE CANIT-DOMAIN-PRO LICENSE

NOTE: ALTHOUGH CANIT-DOMAIN-PRO IS DISTRIBUTED WITH CLAM ANTIVIRUS,WE DO NOT MAKE ANY REPRESENTATIONS AS TO ITS EFFECTIVENESS AT STOP-PING VIRUSES. ROARING PENGUIN HEREBY DISCLAIMS ALL WARRANTY ONTHE ANTI-VIRUS CODE INCLUDED WITH CANIT-DOMAIN-PRO, OR WHICH INTER-FACES TO CANIT-DOMAIN-PRO. WE ARE NOT RESPONSIBLE FOR ANY VIRUSESTHAT MIGHT EVADE A VIRUS-SCANNER INTEGRATED WITH CANIT-DOMAIN-PRO.

11. Disclaimer of Warranty (Time-Critical Mass Mailings)

CANIT-DOMAIN-PRO IS NOT DESIGNED FOR TIME-CRITICAL EMERGENCY MASSMAILINGS. AN EMERGENCY MASS-MAILING MAY OVERLOAD CANIT-DOMAIN-PRO AND CAUSE DELAYS. ROARING PENGUIN HEREBY DISCLAIMS ALL WAR-RANTY ON THE ABILITY OF CANIT-DOMAIN-PRO TO DELIVER MASS MAILINGS INA TIMELY FASHION. IF YOU REQUIRE EMERGENCY MASS-MAILINGS YOU MUSTCONFIGURE THEM TO BYPASS THE CANIT-DOMAIN-PRO FILTER.

M.1 THE CANIT DATA LICENSE

Roaring Penguin makes available certain data that are used by CanIt. This license covers the RPTNBayes data and the Roaring Penguin RBLs. The data are owned by Roaring Penguin and their use islicensed under the following terms:

1. You may update the RPTN data once per day per Roaring Penguin download username. RoaringPenguin reserves the right to cut off downloads if more than one download per day per usernameis attempted.

2. You may use the RPTN data only in conjunction with your properly-licensed CanIt installation.

3. You may not redistribute the RPTN data.

4. If your support term expires, you lose the right to use RPTN data for any purpose whatsoever.

5. You may make use of the Roaring Penguin RBLs from within CanIt. You may not query themwith any other software.

6. You may use the Roaring Penguin RBLs only in conjunction with your properly-licensed CanItinstallation.

7. You may not redistribute the Roaring Penguin RBL data.

8. If your support term expires, you lose the right to use the Roaring Penguin RBLs.


Index

access rights, see permissionsaccount-info, 157active streams, 123address mapping, 83

scenarios, 85wildcards, 84

addresses, locked, 185alias, 50anomaly, 378architecture, 334attachment, 187authentication, external, 137

backscatter, 133backups, 225base realm, 40basic setup wizard, 56Bayes Database, 373

Berkeley, 373CDB, Cluster Considerations, 374PostgreSQL, 373

Bayes journal, 162Bayesian filtering, 161

votingunauthenticated, 161

best practices, 222branding, 79

canit.conf, 336classes

stream, 165cloning, 352cluster communication settings, 341configuration file, 336creating a group, 122creating realms, 40cron job, 106, 347cron settings, 336

customization of theme, 79

data license, 384database settings, 336database, moving, 348debugging logs, 361default stream, 85deleting a group, 122deleting a stream, 125deleting realms, 41delivery status notification, blocking, 133direct queue injection, 74disabling features, 74discarded message, 37disclaimer, 125disk imaging, 352DNS blocklists, 110domain configuration wizard, 227domain mapping, 81

AsIs, 82ChopDomain, 82ChopUser, 82Database, 82Program, 82

domain, locked address, 185download, RPTN, 162downloading logs, 215dump, restoring from, 345

event log, 362expire, 106

non-spam, 106spam, 106

external authentication, 137

false positive, 19features, 74

direct queue injection, 74


386 INDEX

disabling, 74enabling, 74

filter settings, 339filtering outbound mail, 125final stream, 175firewall, 346firewall rules

RPTN, 163flow of mail, 28forwarding logs, 216

geolocation data, 163global settings, 105grantability, permission, 171greylisting, 23, 219group, 122

creation, 122deletion, 122editing, 122

group permissions, 165

hooks, 348HTTPS, 81

inheritance, 173

joe-job, 133

known networks, 65

license, 381data, 384

locked address domain, 185locked addresses, 185logging, 361

events, 362logs, downloading, 215logs, forwarding, 216logs, searching, 209

macros, office, 132mail flow, 28mapping, 50maximum size, 105memcached, 353memory, 343message

status, 35message size, maximum, 105milter, 23MIMEDefang, 23mimedefang settings, 337moving database, 348MX, secondary, 130

newy-seen domains, 133

office macros, 132opt-in, 107outbound mail, filtering, 125ownership and permissions, file, 223

periodic reports, 179permission grantability, 171permissions, 165

granting, 167group, 165stream, 165, 167

permissions and ownership, file, 223phishing, 132phishing URL, 132phishing URLs, 112plus hack, 108post-cron-hook, 347privileges

user, 117root, 117write, 117

program user lookup, 154provisioning, 135proxying

URL, 191

RAM, 343rate-limiting, 69real-time blocklist, 110realm, 24, 39

base, 40creating, 40definition, 47deleting, 41mappings, 41

realm mappings, 41receive-only addresses, 222


INDEX 387

Received: header, 130relay host, 24remailing, 37report, RPTN, 162reports, periodic, 179restoring database, 345rewrite, 157Roaring Penguin Training Network, see RPTNRPTN, 162

firewall rules, 163RPTN download, 162RPTN report, 162RPTN setup wizard, 56rule

copying, 129prioritization, 28

ruleset update, 163

searching logs, 209secondary MX, 36, 130security, 223

network, 224PHP, 224PostgreSQL, 224ssh, 224

Sender Policy Framework, see SPFSender Rewriting Scheme, see SRSSendmail plus hack, 108server, verification, 57settings, cluster communication, 341settings, cron, 336settings, database, 336settings, filter, 339settings, mimedefang, 337settings, storage manager, 341settings, ticker, 340Shortener404, 133simple GUI, 173simple interface, 108Simple Mail Transfer Protocol, see SMTPSMTP, 24SMTP AUTH, 69SMTP authentication, 107SMTP Server Testing, 195SNMP, 367special streams, 175

SPF, 24SRS, 24, 109storage manager, 201storage manager settings, 341stream, 24, 47

active, 123default, 51, 85definition, 47deleting, 125final, 175granting access to, 119inheritance, 173mapping, 50special, 175

stream classes, 165stream permissions, 165, 167streaming, 32

methods, 32AsIs, 32ChopDomain, 32ChopUser, 32Database, 32Program, 33User Lookup, 33

syslog, 361system check, 75

tempfail, 25templates, 76temporary failure, see tempfailTesting, SMTP Server, 195theme customization, 79ticker settings, 340tuning, 342

unauthenticated voting, 161updates, rules, 163URL Proxying, 191URL Shortener, 133URLs, phishing, 112user

adding, 117deleting, 119editing, 118

user lookup, program, 154user lookup, rewrite, 157


388 INDEX

user privileges, 117users, 116

verification server, 57non-standard port, 59

virus, 106voting

unauthenticated, 161

welcome screen, 54wizard, 56

basic setup, 56RPTN setup, 56


canit-domain-pro administration guide

Documents