Table of Contents |
---|
This page was created in April of 2017 to help modernize our system records.
SysAdmins
Goals
KAM: Apache SpamAssassin is a framework for writing rules. I deliver rules to prove the code works but I don't view that the project has to provide rules. I use this as a guidance in where I spend my focus. Beyond that, my goal with belonging to the SysAdmin group is to ensure the project is supported with modern, secure hardware and software with a bus factor greater than one.
...
Read-Write: https://svn.apache.org/repos/asf/spamassassin
Repo | Contents | Notes |
sysadmins | Server and application configs | Encrypt passwords and sensitive information – NEED TO SPECIFY HOW WE WANT TO DO THIS |
dns | Configs and records related to spamasssassin.org | Hosted by PowerDNS on sa-vm1.apache.org as hidden master |
site | http://spamassassin.apache.org site contents |
Bugzilla
...
- Create an account at https://wiki.apache.org/spamassassin using your full name (i.e. Jane Doe).
- Email sysadmins@spamassassin.apache.org to request access to the wiki:
*Contributor only
*Contributor and Admin
NOTE: Write access to the wiki is to anyone who has created a login name on the wiki whose name has been added to the page https://wiki.apache.org/spamassassin/ContributorsGroup
...
- Open an SSH tunnel: ssh -f sa-vm1vm.apache.org -L 8090:localhost:8090 -N
- Open web interface: http://localhost:8090
- Login with admin. (Password is encrypted in sysadmins/accounts.)
Zone | Server | Contact | Notes |
spamassassin.org | ns2.pccc.com | Kevin McGrail kevin.mcgrail@mcgrail.com, kmcgrail@apache.org | Instant updates via NOTIFY |
ns2.ena.com | Dave Jones djones@ena.com, davej@apache.org | Instant updates via NOTIFY |
dns-master.sonic.net | Grant Keller grant.keller@sonic.com | Hidden slave, 5 to 10 min delay of public slaves after NOTIFY |
ns.hyperreal.org | Brian Behlendorf | Currently not used since DJBDNS doesn't support NOTIFY or EDNS over TCP |
Standards
*Apache Infrastructure standard is Ubuntu 16.04 LTS
*Cron entries should be in new standard locations /etc/cron.d, /etc/cron.daily, etc. and avoid using user's crontab
*Custom scripts should reside in /usr/local/bin if they are not direcly related to SpamAssassin processing that should be in /usr/local/spamassassin
*Symlink scripts from /usr/local/bin to /etc/cron.d, /etc/cron.daily, or /etc/cron.weekly. This provides easy discovery and future management by others on the sysadmins team.
*Scripts and cron entries should mail output to the sysadmins mailing list
...
*minotaur.apache.org - handled various build and devel related tasks
*hyperion.apache.org - likely a Solaris box that had backup data of next server
*spamassassin.zones.apache.org - DIED - was replaced with spamassassin-vm
*spamassassin.zones2.apache.org - deprecated by Infra, replaced by sa-vm1.apache.org
*spamassassin-vm.apache.org - deprecated by Infra, replaced by sa-vm1.apache.org
*buildbot, ruleqa, etc. are aliases of above deprecated servers
Servers
Hostname | Function | Software | Configs/Location | Resource/URL | SVN Location |
apachesf. |
sonic. |
net | Donated by Sonic |
CentOS 7 | apachesf.spamassassin.org (64.142.56.146) | ||||
colo.sonic.net | Retired | 76.191.162.2 | |||
trap-proc.spamassassin.org | Retired | a.k.a spam-trap.spamassassin.org (192.87.106.247) | |||
sa-vm1.apache.org | DNS Hidden Master | PowerDNS | / |
sa-vm1.apache.org
DNS Hidden Master
PowerDNS
etc/powerdns/pdns.d/pdns.local.conf | spamassassin.org | dns (webserver API key redacted) |
Rsync Mirrors | rsyncd | /etc/rsyncd.conf | rsync.spamassassin.org | trunk/build/automc/etc-rsyncd.conf |
Web Server | apache2 | /etc/apache2/sites-available/automc.conf | updates.spamassassin.org | trunk/build/automc/automc-apache2.conf |
apache2 | /etc/apache2/sites-available/automc.conf | ruleqa.spamassassin.org | trunk/build/automc/ruleqa.cgi | ||
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id=" |
1d53d356- |
108f- |
4add- |
b325- |
d37f2551dcd3"><ac:plain-text-body><![CDATA[ |
SaUpdateMirrorSetup | svn | rsyncd [updates] for mirrors | spamassassin.apache.org/updates | [site/updates | http://svn.apache.org/repos/asf/spamassassin/site/updates/]/MIRRORED.BY | ]]></ac:plain-text-body></ac:structured-macro> |
Nightly Masscheck | cron/scripts | /usr/local/spamassassin/automc/rsync/tagged_builds | ruleqa.spamassassin.org | trunk/backend/nitemc/README |
RuleQA web UI | cron/scripts | /usr/local/spamassassin/automc/html | ruleqa.spamassassin.org |
Backups
An old backup exists in sa-vm1.apache.org:/usr/local/spamassassin/backups/spamassassin-vm. It's a large bzip'd tar file so make sure you don't extract it and fill up the filesystem.
...
Wiki Markup |
---|
*25 2 \* \* \* automc *~/svn/trunk/build/mkupdates/do-stable-update-with-scores* **~/svn/masses/rule-update-score-gen/do-nightly-rescore-example.sh* **~/svn/masses/rule-update-score-gen/generate-new-scores.sh* *uses ~/tmp/generate-new-scores for SVN work area *sorts out the usable corpus from the latest 'SVN revision' at the top of the submitter's log file which should match the latest tagged build of SVN rules *checks $\{REVISION\} LINE 123 NEEDS IMPROVEMENT!!! THIS SVN REVISION NEEDS TO BE CLOSELY TIED TO THE REVISION THAT WAS STAGED IN THE MASSCHECK RSYNC DIR. *checks the sorted corpus for a minimum number of valid contributors and ham/spam **~/svn/trunk/build/mkupdates/mkupdate-with-scores* *uses ~/tmp/sa-mkupdate for SVN working area *gets latest SVN $\{REVISION\} from rulesrc/scores/score-set\* *masses \-> perl Makefile.PL && make (complete build of SA and test) *perl hit-frequencies *garescorer - compiles and runs it, requires build/pga *sends email if not enough masscheck submitters or usuable ham/spam for the latest SVN revision *creates $\{REVISION\}.tar.gz $\{REVISION\}.tar.gz.sha1 and $\{REVISION\}.tar.gz.asc in /var/www/automc.spamassassin.org/updates for mirrors to pull *updates DNS TXT entries \[0-3\].3.3.updates.spamassassin.org and 0.4.3.updates.spamassassin.org -- versions >= 3.4.1 have a CNAME to 3.3.3.updates.spamassassin.org *Script rewrite notes: *Make each primary step modular since these steps are commmon in other scripts *Should check for minimum contributors of ham/spam up front and not waste resources if requirements not met *These 3 scripts above all share the same temp working dir. This should be determined from config file or relative path of user's home dir for flexibility. *Should be able to run the ham/spam processing in parallel and merge the results together to cut this time in half *Temp working dir for the corpus should be persistent so the rsync copy will be faster. *Usuable corpus symlink setup could be improved. Invalid stale corpus should be removed into an archive/excluded dir. |
...
Wiki Markup |
---|
*30 8 \* \* \* automc *~/svn/trunk/build/mkupdates/run_nightly* > /var/www/automc.spamassassin.org/mkupdates/mkupdates.txt *Currently $\{SA_VERSION\} = "3.4.2" *$\{REVISION\} = latest SVN revision THIS NEEDS TO BE ADDRESSED!!! NEED TO PREVENT REVISION FROM MESSING UP THE MASSCHECK PROCESSING. *creates new rules/active.list *commits new rules/active.list *runs spamassassin lint against the updated rules and checks in a tagged version of 'sa-update_$\{SA_VERSION\}_$\{TSTAMP\}' *commits "promotions validated" and emails dev@spamassassin.apache.org *creates $\{REVISION\}.tar.gz $\{REVISION\}.tar.gz.sha1 and $\{REVISION\}.tar.gz.asc in /var/www/automc.spamassassin.org/updates for mirrors to pull *updates DNS TXT entries \[0-3\].3.3.updates.spamassassin.org and 0.4.3.updates.spamassassin.org -- versions >= 3.4.1 have a CNAME to 3.3.3.updates.spamassassin.org *Script rewrite notes: *Uses many of the same primary steps previous section so reuse the code and not have to maintain multiple versions *Should be turned into generic script that can be run on demand via SVN trigger/polling |
...