gnu mailman implementation at argonne nlit 2009 – knoxville, tn – june, 2009 barry s. finkel...
TRANSCRIPT
GNU Mailman Implementation at Argonne
NLIT 2009 – Knoxville, TN – June, 2009
Barry S. Finkel
Computing and Information Systems Division
Argonne National Laboratory
630-252-7277
2
Central Mailman Server
Computing and Information Systems (CIS) runs the central Mailman server for ANL.
Several scientific divisions have their own servers and mailing list managers tailored to meet their own needs.
3
Requirements
Web interface for managing listsWeb interface for viewing list archivesNeeds to understand attachments and
encoded e-mail
4
Evaluation of Mailing List Managers
Ecartis (free, open-source)Mailman (free, open-source)Listserv ($, commercial product)Lyris ($, commercial product) Majordomo (free, open-source, no longer
supported)Majordomo 2 (free, open-source, small user
base and development)
5
Evaluation Results (1)
People preferred Mailman over Ecartis.No money for ListservLyris is good for marketing campaigns.Majordomo lacks features
– No web interface– Web archives via external application– Archives not integrated with list restrictions– Majordomo perl scripts do not handle
encoded mail.
6
Evaluation Results (2)
•Subscriptions, when sent in quoted-printable format, become [email protected]=20 .
– We easily cannot limit subscriptions to, e.g., anl.gov without list manager approval of all subscriptions.
– Administrators cannot manage their lists except via e-mail – prone to error.
Majordomo 2– Still under development– Not as robust as Mailman
7
History
Intended to replace MajordomoMajordomo is still used for one-time
specialized lists built just before use.One big Majordomo list still used – the owner
does not want passwords to unsubscribe.Neither Majordomo nor Mailman is used for
small mailing lists (e.g., computer alerts, etc.).– We do not need archives.– We don’t want the overhead of routing mail
to another machine and back.– Lists reside on the @anl.gov alias machine.
8
Types of Lists at Argonne (1)
Standard lists with subscribes/unsubscribesTransmittal lists – documenting system
changes– Subscribes were handled by a flat file.– Majordomo did the archiving.
– We used WAIS to search the archives.
9
Types of Lists at Argonne (2)
Lists where subscribes come from external DB– Human Resources (HR) Database•Labcast/DOEcast lists for each ANL division/group•Argonne-today-employees (non opt-out)•Lists for some buildings•Lists for some division/building pairs
– ISAAC (Information Services Authentication and Access Control)
10
Types of Lists at Argonne (3)
Lists where the subscription list is built from external sources just before use. – Example: all employees in the PPO medical
plan– Many of these are being converted to
individual Mailman lists.
11
DOEcast/Labcast (1)[email protected] goes to three addresses, two
in Communications and Public Affairs, and one in CIS.
If C&PA decides that the doecast should be distributed, they send it to [email protected].
We have four labcast lists:– null (the three addresses above)– employees– entiresite (includes Credit Union, Security
Force, DOE, NCI, Omega Travel, etc.)– entiresite-plus-suu (APS Users)
12
DOEcast/Labcast (2)
Via a web page C&PA activates one of the lists.
They send the mail to [email protected] reset the list back to the null list.The labcast list expands to one of three lists
that contain the various divisional Mailman mailing lists.
When a new division acronym occurs in HR, I build a new Mailman division list.
13
Hardware and Operating System (1)
Sun Fire X4200 M1Ubuntu amd64Decided we did not need a redundant server
configuration.
14
SourceForge vs. OS Package Install (1)
Package built at ANL from SourceForge source– Debian/Ubuntu package had too many
patches.– Many changes in Ubuntu were
undocumented.– One patch removes a needed library.– I wanted to know exactly what code was
running.
15
SourceForge vs. OS Package Install (2)
– I kept one patch – puts libraries in the proper directories for Ubuntu.
– I reviewed latest Ubuntu package for the jaunty release and found same problems (fewer patches, one is wrong)
Built packages for Mailman 2.1.9 and 2.1.11Built/testing new package for Mailman 2.1.12
16
New Mailman “Home” Page (1)
Mailman-generated home page is not user-friendly.– All advertised lists are presented alphabetically.– No list categories– Hard to find a list if name is unknown
Categorize each list– “{CATEGORY}” at end of each list description– “{CAT1,CAT2}” if multiple categories
Change mailman/listinfo URL to a new name so that the Mailman-generated page is available, if needed.
17
New Mailman “Home” Page (2)
XML file re-built every 20 minutes.– Use “link” to retrieve the renamed Mailman-
created page.– Parse this page in conjunction with the
category string list.CIS Web Group creates a dynamic “home”
page based on this XML file.Not accessible from off-site. I might want to
have an external page created.
18
New Mailman “Home” Page (3)
19
New Mailman “Home Page (4)
20
List Classifications (1)
Posters– {An} Anyone– {Bn} Members only– {Cn} Members + selected others– {Dn} Selected others only– {En} Only anl.gov e-mail addresses (includes
uchicago.edu and ch.doe.gov) Subscriptions
– {X1} open– {X2} closed– {X3} closed + closed unsubscribes
21
List Classifications (2)
List configuration templates– I have 15 Xn.config files on the Mailman machine.– I also have {building,labcast}.config files.– I also have some miscellaneous config files.– Total of 25 template configuration files
22
Creating a New List (1)
I do not create lists via the web interface, as the web creates a list with default parameters.
To create a new list named NEWLIST:– Copy a template configuration file.– vi NEWLIST.config– Change certain list parameters:•Owner(s)•Description (includes the “{category}” string)• Info• Acceptible_aliases (@anl.gov + @lists.anl.gov)• Archive? Yes or no? Public or private?•Digestable?
23
Creating a New List (2)
– ./create_list.exec NEWLIST•E-mail address of one owner•List password
– Add the list name to the /etc/mail/aliases.template file on the @anl.gov alias server Postfix machines. All lists are known as [email protected] and [email protected] to avoid conflict with a regular [email protected] e-mail alias.
24
Wiki Pages
Documentation for list usersDocumentation for list owners
– I highlight those that the owner might want to change.
– Includes documentation on all the list parms– Form for requesting a new Mailman list •Requestor selects list parameters. •Requstor sends e-mail to the help desk.
Status – items to do and those completed Location of all of the Mailman files in Ubuntu
25
Don’t Crawl the List Archives Nor the List Admin Pages.
mailman% cat /usr/lib/cgi-bin/mailman/robots.txt
# This robots.txt file is for Mailman on the lists.anl.gov machine.
# 30Mar07 0235PM Barry Finkel
#User-agent: *
# /pipermail/ contains the archives.
Disallow: /pipermail/
# /mailman/admin contains the administrative links.
Disallow: /mailman/admin
# /mailman/private/ contains links to the private archives.
Disallow: /mailman/private
mailman%
26
External Lists
There are only a handful of ANL lists that are accessible outside ANL.
I block access to lists from offsite and enable specific lists for offsite access via /etc/apache2/sites.d/lists.anl.gov .
The Mailman listinfo home page is not accessible from off-site.
27
Populating Lists from HR and ISAACList files are placed on a set schedule (different
directories for each source).Via cron, I do the following:
– “Compare” the file with the local file (ignoring comment lines).
– If different, then•Copy the new file to the local directory.•Run sync_members to update the Mailman list. All updates are “silent”.
– Any list can have a LISTstatic file containing additions to the list.
28
Lists of Members
I run a cron every hour that lists most of the list memberships.
The file is placed in AFS for CIS IT personnel use.
29
Bounce Processing
I run a report every morning to report the subscriptions with bounce_score > 0
http://veenet.value.net/~msapiro/scripts/get_bounce_info.py
I check the output every morning to spot bad e-mail addresses, especially in the argonne-today-employees list.
Prior to Mailman 2.1.10 the report had old information along with the current bounce information.
Lists with varying posting frequencies require tuning of the bounce processing parameters.
30
Mailman Password Issues
Most users do not need to know their Mailman list passwords.
All auto-subscribes are silent; no e-mail is sent.No list monthly password reminders are sent.Any user can retrieve his/her list password via
the list home page• to see the list archives, • to unsubscribe, or • to set digest/nodigest (or other options).
31
List Configuration Trends and Analysis
On the first of each month I make a snapshot of each list configuration.
I compare to the previous month’s snapshot to see what changes have been made.
Can be used as a backup in case a list administrator destroys a list configuration
I can use “grep” to find lists with a given configuration, as all list configuration files are in the same directory.
32
A Useful List
I have a list mailman-list-owners (populated automatically) to communicate with all of the list owners.– Updates– Education– Features– Desires
33
Problems with Mailman
Multi-part messages can be corrupted:– Signed mail – list footer changes the mail
and breaks the signature. – List footer also causes problems with some
HTML-formatted mail.Mostly caused by use of footers or headers in
message
34
Problems with Ubuntu/Apache/BlackBerry
My sync_lists script infrequently produces “/usr/bin/cut: not found”. Same for tr and awk. Counts this year: cut - 8, tr - 5, awk - 0 Counts for 2008: cut - 14, tr - 11, awk - 1
Apache2 produces error messages for Mailman web page accesses that succeed. I cannot determine what messages in the logs represent real errors.
“Reply-all” on a BlackBerry will result in mail to listname and listname-bounces. We are waiting for a response from RIM.
35
Futures
Mailman 2.1.12– Installed on my test virtual machine– Need to install a Mailman patch and test.
New searchable archiver to replace Pipermail; have done little research and no testing
Need a web page where someone can request a new list (to replace my wiki page)
Maybe a new “home” page for externally accessible lists
We need a way to do Mailman command-line tasks from a machine external to Mailman.
36
Questions?