Difference between revisions of "Moving off Google Apps"

From Wiki | LUG@UCLA
Jump to: navigation, search
 
(32 intermediate revisions by 7 users not shown)
Line 1: Line 1:
LUG@UCLA plans to move all mail services (including lists) off Google apps. This is a long term project, but the ETA is before Spring quarter 2014.
+
LUG@UCLA plans to move all mail services (including lists) off Google apps. This is a long term project, but the ETA is before Summer 2014.
  
 
== Design ==
 
== Design ==
  
KISS. Try to use the least amount of components, and don't overcomplicate the configuration. For example, don't use maildrop if Dovecot already has an MDA/LDA.
+
KISS. Try to use the least amount of components, and don't overcomplicate the configuration. For example, don't use Maildrop if Dovecot already has an MDA/LDA. Don't use the high-performance sdbox format if Maildir is well supported and tested.
  
* MTA: Postfix
+
=== Overview ===
 +
 
 +
* MTA: Exim
 
* MDA/LDA: Dovecot LDA
 
* MDA/LDA: Dovecot LDA
 
* MSS: Dovecot
 
* MSS: Dovecot
Line 11: Line 13:
 
* lists: Mailman
 
* lists: Mailman
 
* storage format: Maildir
 
* storage format: Maildir
* storage redundancy: Tahoe-LAFS
+
* storage backup: cron + duplicity to LUG Lounge, cron + duplicity to VTLUUG, etc.
* storage backup: duplicity to NFS share, duplicity to VTLUUG, etc.
+
  
=== Online storage ===
+
=== Mail transfer ===
  
Use the traditional Maildir format for storing mail. Make the MDA/LDA store it under the Tahoe-LAFS $BASEDIR directory so it can be accessed from multiple different MSSs and MUAs.
+
* set up two MTA servers (''at most'' one in the lounge), each with its own MDA and MSS.
 +
* configure BIND to broadcast two mx records (one primary, one secondary)
 +
* use an SPF record in DNS configuration
 +
* tutorial - exim on debian: https://wiki.debian.org/Exim
 +
* tutorial - dkim with exim: http://atmail.com/kb/2008/installing-dkim-for-outbound-messages/
  
=== Offline storage ===
+
=== Mail delivery ===
  
Occasionally copy the Maildir directory out of the Tahoe-LAFS share since we don't actually trust Tahoe-LAFS. We respect people's privacy, so don't just rsync it out to a 3rd party. Easiest solution would be to use Duplicity to automatically perform encrypted, incremental backups to the 3rd party.
+
* configure one MDA for each MTA server
 +
* MDA shall deliver to a Maildir (to be backed up)
 +
 
 +
==== Online storage ====
 +
 
 +
* Make the MDA store the Maildir on:
 +
*# the bare filesystem, or
 +
*# under a mounted clustered directory (e.g. glusterfs)
 +
 
 +
==== Offline storage ====
 +
 
 +
Occasionally copy the Maildir directory out of the clustered share since we don't actually trust online solutions. We respect people's privacy, so don't just rsync it out to a 3rd party. Easiest solution would be to use Duplicity to automatically perform encrypted, incremental backups to the 3rd party.
 +
 
 +
=== Access ===
 +
 
 +
* MSS will provide POP3 and IMAPS access
 +
* MSS authentication/authorization:
 +
** use LDAP+Kerberos for lookup/authentication/authorization '''(for linux.ucla.edu emails only)'''
 +
*** [http://wiki2.dovecot.org/HowTo/DovecotOpenLdap Dovecot + LDAP tutorial]
 +
** manually configure MSS credentials in /etc/dovecot/users.conf '''(for other domains (e.g. acm.ucla.edu))'''
 +
* configure multiple LDAP+Kerberos instances for each MSS.
 +
* periodically and automatically replicate LDAP directory and Kerberos principals across all servers.
 +
** [http://www.openldap.org/doc/admin24/replication.html LDAP replication tutorial]
 +
** [http://www.tldp.org/HOWTO/Kerberos-Infrastructure-HOWTO/server-replication.html Kerberos replication tutorial]
 +
 
 +
ports:
 +
* POP3 over SSL: 995 tcp/udp
 +
* IMAP over SSL: 993 tcp/udp
 +
* HTTP over SSL: 443 tcp
  
 
=== Transitional details ===
 
=== Transitional details ===
  
 
* How to migrate emails from Google Groups to Maildir readable by Mailman?
 
* How to migrate emails from Google Groups to Maildir readable by Mailman?
* How to migrate users of @linux.ucla.edu emails to the internal system (e.g. login access to POP3/IMAP/Roundcube)? Make use of LDAP?
+
** fetch all mails using fetchmail, dump into Maildir.
* How to notify subscribers to the mailing list? Do they need to know?
+
** delete all my personal mails that got pulled in.
 
+
**
 +
* How to migrate users of @linux.ucla.edu emails to the internal system (e.g. login access to POP3/IMAP/Roundcube)?  
 +
** look for a way to export a list of users from Google Apps.
 +
** make use of LDAP/Kerberos to authenticate.
 +
**
 +
* How to migrate subscribers to the GNU Mailman mailing list?
 +
** export a CSV list of users from the Google Groups members page.
 +
** grep/sed the list for the following information: Full Name, subscribed email,
 +
**
 +
 
== For users with LUG emails ==
 
== For users with LUG emails ==
  
 
== For subscribers to the mailing lists ==
 
== For subscribers to the mailing lists ==
 +
 +
== External links ==
 +
 +
* http://shearer.org/MTA_Comparison

Latest revision as of 23:27, 25 February 2014

LUG@UCLA plans to move all mail services (including lists) off Google apps. This is a long term project, but the ETA is before Summer 2014.

Design[edit]

KISS. Try to use the least amount of components, and don't overcomplicate the configuration. For example, don't use Maildrop if Dovecot already has an MDA/LDA. Don't use the high-performance sdbox format if Maildir is well supported and tested.

Overview[edit]

  • MTA: Exim
  • MDA/LDA: Dovecot LDA
  • MSS: Dovecot
  • MUA: Roundcube
  • lists: Mailman
  • storage format: Maildir
  • storage backup: cron + duplicity to LUG Lounge, cron + duplicity to VTLUUG, etc.

Mail transfer[edit]

Mail delivery[edit]

  • configure one MDA for each MTA server
  • MDA shall deliver to a Maildir (to be backed up)

Online storage[edit]

  • Make the MDA store the Maildir on:
    1. the bare filesystem, or
    2. under a mounted clustered directory (e.g. glusterfs)

Offline storage[edit]

Occasionally copy the Maildir directory out of the clustered share since we don't actually trust online solutions. We respect people's privacy, so don't just rsync it out to a 3rd party. Easiest solution would be to use Duplicity to automatically perform encrypted, incremental backups to the 3rd party.

Access[edit]

  • MSS will provide POP3 and IMAPS access
  • MSS authentication/authorization:
    • use LDAP+Kerberos for lookup/authentication/authorization (for linux.ucla.edu emails only)
    • manually configure MSS credentials in /etc/dovecot/users.conf (for other domains (e.g. acm.ucla.edu))
  • configure multiple LDAP+Kerberos instances for each MSS.
  • periodically and automatically replicate LDAP directory and Kerberos principals across all servers.

ports:

  • POP3 over SSL: 995 tcp/udp
  • IMAP over SSL: 993 tcp/udp
  • HTTP over SSL: 443 tcp

Transitional details[edit]

  • How to migrate emails from Google Groups to Maildir readable by Mailman?
    • fetch all mails using fetchmail, dump into Maildir.
    • delete all my personal mails that got pulled in.
  • How to migrate users of @linux.ucla.edu emails to the internal system (e.g. login access to POP3/IMAP/Roundcube)?
    • look for a way to export a list of users from Google Apps.
    • make use of LDAP/Kerberos to authenticate.
  • How to migrate subscribers to the GNU Mailman mailing list?
    • export a CSV list of users from the Google Groups members page.
    • grep/sed the list for the following information: Full Name, subscribed email,

For users with LUG emails[edit]

For subscribers to the mailing lists[edit]

External links[edit]