Website:News

=News=

January 13, 2012
Our mail server, hermes will be down for a few hours tonight (1/13/2012) for upgrades that we believe will improve service. No received mail will be lost, but queued during that period and delivered when the upgrades are complete.

December 28, 2011
The UGCS sysadmins would love constructive feedback about our services. Feel free to email us as sysadmins at ugcs dot caltech dot edu.

December 12, 2010
Earlier this morning, mail delivery was held up for several hours. The mail queue has now cleared, and all messages sent should be delivered.

November 16, 2010
We will have a maintenance window on Saturday, November 20 from 9am to 4pm. During this time, various UGCS services may be down, and we will be restarting all login servers.

October 15, 2010
We have finished the rollout of Debian Squeeze on all of our shellservers and mortals. We think we have a complete development environment for most languages on them- but if you find something not there that you want, let us know and we'll get it installed.

September 13, 2010
We are in the proccess of moving our shellservers and mortals to a more recent version of Debian ("squeeze"), so please let us know if there are any bugs that need to be fixed. Login may be unstable for the next day or two as we reboot the machines to load the new images, but they should be stable after that.

August 13, 2010
One of our AFS servers (apollo) failed to reboot around 11am, causing an AFS outage. Mail bounced for about 20 minutes, but none was silently dropped. As of 12:30pm, things are on their way back up.

August 8, 2010
Air conditioning failure in the UGCS server closet caused a number of difficulties. Apollo crashed, taking the AFS root volume with it, and so homedirs, the website, and mail were temporarily unavailable. Furthermore, the mortals subcluster was turned off to save heat. All mail should have been now delivered, and homedirs should now be available. For further information or questions, email the sysadmins.

June 21, 2010
An unexpected power outage in Winnett from 2:00 to 6:00 AM resulted in massive service disruption (aka the whole cluster was down) for about four hours. Any mail sent during that time was collected offsite and has since been delivered. All services and servers should now be up and running. If you have any questions, please email the sysadmins.

June 17, 2010
We've moved all the shellservers to a new 64-bit image, and the lab has been cleaned and reorganized. There are some small changes to the new image that need to be made, so if you notice anything out of place please email the sysadmins.

May 23, 2010
We now have two machines, terpsichore and minthe, that have high-performance graphics cards (Nvidia GTX 260), as well as 4gb ram and a quad-core processor. These computers were purchased with a grant from the Housner Fund

April 13, 2010
Office hours this term will be be Mondays from 10-11:30pm.

March 22, 2010
We will have some downtime on Monday, March 28 starting at 12:01am and lasting until 6am at the latest.

January 6, 2010
Office hours for this term will start next week and run from 7:30-9pm.

December 2, 2009
Office hours tonight will run from 7:15-8:00pm.

November 4, 2009
The Mortals cluster is back up.

October 18, 2009
We had some brief downtime on posiedon, our webserver, this morning from 9:30 to 11:30. All services should be back up- please let us know if you have problems.

October 6, 2009
We are going to start holding Office hours in UGCS. Current office hours times are Wednesdays 7:30pm - 9pm, but are subject to change- see the page for updates. Come by to say hi, get your password reset, or ask questions about Linux or UGCS.

September 27, 2009
We are currently migrating to Postgres 8.3. For portions of tonight, postgres may be down, and changes made may be lost.

Update: 11:59pm The migration should be complete, and databases should connect to 8.3 by default.

September 13, 2009
We will be migrating to Postgresql 8.3 in a few weeks. To help you make this migration, we have set up a Postgres 8.3 server at postgres:5433. Please test your applications against this new database. See Postgres migration for more information. As always, contact us if you have any questions.

August 22, 2009
A "phising" email was just sent to many UGCS users which asks you to reply with a username and password. Do not reply to this message- it is a scam. UGCS sysadmins will never ask you for your password.

August 19, 2009
We have set /afs/.ugcs/ugcs-admin to be world-readable. You can now look through the various utilites we have written to help automate the cluster administration. Over the next few months, we will be setting up a website to better document the code we have written.

We also have a read-only copy of our svn repository available at http://www.ugcs.caltech.edu/svn, or you can access it at file:///afs/.ugcs/ugcs-admin/svn We're quickly adding our projects to it, so check back often!

August 14, 2009
Over the next day or two, we will be rebooting our shellservers with a new setup. This will improve their speed and usability, but will result in some downtime in various shellservers. Contact us if you're planning on running and long-term jobs. $$Insert formula here$$

June 21, 2009
The outer door has been fixed by "facilities".

June 20, 2009
The outer door to UGCS is jammed closed, making the lab inaccessible. If you are trapped inside, please send us an email or call security at 626-395-5000 (extension 5000 on campus). Otherwise, the lab will not be available for a few days. We will work with Facilities to take care of this as soon as we can.

May 5, 2009
UGCS is running its annual sysadmin search. If you are interested in being a UGCS syadmin, please take our sysadmin survey.

April 3, 2009
All user principles have been modified to allow Kerberos tickets to be requested for up to 365 days. Tickets may be renewed up to a total length of 3652 days. Unfortunately, Kerberos does not allow for an unexpiring ticket, analogous to an SSH keypair, but these should be close enough.

Febuary 18, 2009
UGCS now supports svn protocol for subversion source control hosting. We may add additional source control programs if users express interest (eg, git). For more information, see the page on Subversion.

Additionally, we are now running a PGP keyserver, for those of you who make use of such encryption. You can find it at [], [], and [].

October 29, 2008
We will have a UGCS sysadmin meeting on Sunday, November 2, at 5pm in the UGCS lab. All are welcome to attend. See the UGCS Blog for more information.

July 28, 2008
We had an unplanned mail outage last night, and mail was not being delivered. We have fixed the problem and the mail queue should finish flushing with a few hours.

May 22, 2008
There will be scheduled downtime due to Caltech needing to bring down power for a few hours on Friday, May 23 from 4:00 AM - 6:00 AM. We will restore all systems as soon as possible after this event. Our UPS is kind of wimpy, sorry.

May 21, 2008
UGCS is back up after a several-hour power outage. All services should be back to normal- please let us know if you have any problems.

May 13, 2008
Due to a recently discovered vulnerability in OpenSSL, all UGCS host keys will be changing shortly. We will post a GPG-signed list of the new keys when we are done. It is a good idea to download this list and add it to your ~/.ssh/known_hosts file to prevent man-in-the-middle attacks.

Update: Host keys for our login machines have been changed. A GPG-signed list of the key fingerprints is available at http://www.ugcs.caltech.edu/ssh-host-keys

May 13, 2008
Several UGCS, alumni, and IMSS users have been reporting increased phly phishing attempts (IE, trying to get you to send them your username and password and claiming to be the admins). Just a reminder that the UGCS sysadmins will never ask you for a password. If we want to confirm your identity we may send you a long string of letters and numbers and then ask you to send it back.

If you're more paranoid, all official email from sysadmins will be signed by that sender's GPG public key. If we forget, ask for a resent signed version before taking any action. You can find the UGCS Sysadmin key as well as our individual keys at the CA page.

May 11, 2008
The Mortals sub-cluster has been added. Enjoy!

March 18, 2008
We have blocked all Kerberos traffic that does not come from UGCS to help mitigate a recent Kerberos vulnerability.

Update We have patched the vulnerability and non-UGCS Kerberos has been unblocked. Jdhutchin@ugcs.caltech.edu 13:14, 18 March 2008 (PDT)

March 6, 2008
Due to unexplained issues with hera (our main LDAP server), we're routing all LDAP requests through to zeus (the Kerberos head and LDAP backup). As a result, things may be slower and less reliable. We are investigating the issue and hope to have it fixed soon.

March 2, 2008
Like Linux? Want to help make UGCS a better place? Think you've got what it takes to be a UGCS sysadmin? Well, then you're in luck. UGCS is conducting its annual sysadmin search. If you're interested, please take our Sysadmin Survey and submit it by the end of spring break (March 30).

February 21, 2008
In order to complete an upgrade to our mail server, mail services will be down for about an hour beginning around midnight.

Update Mail delivery is back up Jdhutchin@ugcs.caltech.edu 01:22, 22 February 2008 (PST)

February 13, 2008
We have automated the mailing list creation process. To create a mailing list, login to UGCS, and run the command create_mailinglist . Your list will be created within the minute. If you should run into any problems email me. alexr@ugcs.caltech.edu 18:15, 13 February 2008 (PST)

February 10, 2008
CGI scripting and PHP has been disabled due to a serious new kernel vulnerability. We are working on fixing this so that that we can re-enable scripting on our webserver.

Update: Scripting on poseidon has been re-enabled. Jdhutchin@ugcs.caltech.edu 15:17, 10 February 2008 (PST)

February 5, 2008
We are currently experiencing unknown problems with mail delivery. You can still access your mail on UGCS, but our server is currently not accepting new mail.

Update: Mail delivery is working again Jdhutchin@ugcs.caltech.edu 03:38, 5 February 2008 (PST)

January 4, 2008
Happy New Year! Unfortunately, we experienced difficulties with our virus scanning and spam blocking systems that resulted in mails being queued for delivery between January 2 and January 4 being held. We are currently clearing the backlog and all mails should arrive at their destinations shortly; no mail was lost.

Update: The mail queue finished flushing by 9pm. All mail has been delivered and mail delivery is back to normal.

November 4, 2007
The maintenance scheduled for the evening of November 3 has been completed successfully, resulting in greatly increased performance on AFS transfers, mail delivery, and LDAP queries. Additionally, we have deployed a new application security framework in order to proactively detect and block SQL injections and compromised guestbook/bulletin board applications. mod_security may have side effects for your CGI applications, so please don't hesitate to contact us if your scripts stop working.

November 1, 2007
We are planning some downtime for the afternoon and evening of Saturday November 3 in order to install gigabit ethernet cards in our mass fileservers and bring additional backup servers online. There may be disruptions to your ability to access files in your home directory, although we will try to minimize impact as much as possible.

Update: the downtime should be largely finished, except for rolling reboots on our shellservers. Apologies for the disruption to file and e-mail services, a few things didn't go as planned, but everything is back up now, and performance should be vastly enhanced. --Elizabeth@ugcs.caltech.edu 21:32, 3 November 2007 (PDT)

October 25, 2007
2:20am: We appear to be under an smtp DDoS attack. We are taking measures to mitigate this attack. To reduce ldap load, our ldap database has been set readonly, so any changes to user information will not work. Also as a preventative measure, we have added more restrictive filtering rules for our core servers- including blocking non-Caltech Kerberos requests (this will be unblocked in the morning). If you notice that we have blocked a critical service, please contact us. We will post updates here as we have them Jdhutchin@ugcs.caltech.edu 02:25, 25 October 2007 (PDT)

Update All services should be back to normal   Jdhutchin@ugcs.caltech.edu 11:13, 25 October 2007 (PDT)

October 24, 2007
The new UGCS job posting system is up. New jobs postings should be added to /afs/ugcs/drop/jobs. If you want a listing removed or changed, please contact a sysadmin. More documentation on the job system can be found here.

October 24, 2007
We recently needed to restart a large number of services due to the openssl 0.9.8g update. If you notice any problems, please contact us. Additionally, we experienced 3 hours of website homedir downtime on the afternoon of the 23rd due to an AFS readonly mirror bug. We have disabled the readonly mirror behavior until we can investigate and things appear to be stable for now.

October 20, 2007
Our new shell servers are now up and running. Please use to.ugcs.caltech.edu to log instead of logging into calliope. If you need to use GSSAPI authentication with to.ugcs.caltech.edu, you will need to follow the Kerberized SSH walkthrough to enable the GSSAPITrustDns option.

October 15, 2007
We have recently fixed the issue with Apache directory listings and believe that we have something very similar now up and running. The cluster is once again at full function, following unplanned downtime that occurred on October 13 due to power disruption.

October 13, 2007
We lost power and network to the cluster this morning due to some miscommunication with physical plant about the extent to which a power maintenance event would impact us. We are still scrambling to get everything up and working again, fingers crossed Update Services are up and running again. We have also added more shell servers to the to.ugcs rotation.

October 11, 2007
We will be doing a considerable amount of work on the cluster on Saturday, October 13. We do not expect any major downtimes; however, some login servers will be taken offline for short periods of time, and core servers may be rebooted causing brief interruptions. If you notice that something has been down for more than 5 minutes on Saturday, please contact us

October 3, 2007
All home directories were transferred as of Monday, October 1. Some files may have incorrect ownership since the copy was performed as root. If you have any files that were not transferred, or if you need to access the backups of UGCS3 files, please contact the sysadmins and we will copy your files from backup.

September 30, 2007
Home directories are now being copied into place; the process is about 30% complete but will run automatically (although a few users ran out of quota while I was copying and I'll need to reset their quotas and retry the transfer of their homedirs). There will be duplication of a lot of files - in particular, I've preserved your mail and www and cgi trees despite their having already been copied once before to a different location. To decrease your quota usage, you should delete any files you do not need immediately. Thanks for your patience!

September 28, 2007
All UGCS3 services have been shut down and transferred to UGCS4 servers, which are all now in the racks in the server closet. Home directories have been archived and will be shortly copied to the new cluster home directories. We do not anticipate any further major downtime.

If you still need to migrate your password, please use the link to the left. If you've forgotten your password, please e-mail sysadmins@ugcs.caltech.edu - if you used SSH key login previously, we will put up your SSH key on a machine that will use your key to authenticate you and allow you to set a password on the new cluster.

Thanks very much for your patience, and we look forward to serving you.

Your UGCS Sysadmins

September 26, 2007
Home directories on UGCS3 will be frozen in the evening of September 27 in preparation for mass copying to the new fileservers. We are planning to bring down all services beginning as early as the afternoon of September 28 in order to rackmount all of the new hardware, carry out IP address reallocations, and reimage old core servers as user-accessible shell servers in UGCS4. The migration should be fully complete by the end of Saturday September 29.

September 22, 2007
Please make sure to read the Migration FAQs as they answer a number of topics related to the impact on user services during the course of the migration.

September 13, 2007
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1

Dear UGCS users,

We are proud to announce that we are about to complete the rollout of UGCS 4.0, which will offer improved performance, features, and quotas. However, these changes will require some action on your part, as well as awareness that many of the quirks of UGCS behavior you are used to will no longer be present. If you have any questions, comments, or concerns, please leave us an e-mail at sysadmins@ugcs.caltech.edu and we'll respond as soon as we can.

Bottom line
services will be unavailable on that day. will need to log into https://hermes.ugcs.caltech.edu/password.html with it to access mail, and to use the rest of the cluster when the migration is complete. very soon (tentatively September 16); you will need to verify your password first as stated above. (except one-member lists) will be transferred as they are in their state as of September 12. Kerberos for passwordless authentication. home directory. You will not be able to set per-file permissions, only per-directory permissions. The main portion of your home directory will be not readable by anyone other than yourself; in order to share files with other users, you will need to place them in the public subdirectory of your home folder. e-mail and we will provide login instructions for our test machines.
 * We are targeting Saturday September 29 for the main switchover. All
 * Your password is frozen in its current state as of September 12. You
 * Your mail will only be accessible via secure IMAP and POP3, effective
 * /ug/drop/mail is no longer writable, and all existent mailing lists
 * SSH keys will no longer work after the migration; we recommend use of
 * After the migration, your home directory will be copied to your new
 * If you wish to help us beta test the new system please send us an

Authentication
We are migrating from NIS, which stores crypt passwords, to Kerberos; since crypt is irreversible and Kerberos requires a copy of your secret to create your principal, we cannot directly perform this migration for you. You will need to enter your old password and a new password into an online form (using SSL). The application will then enable your kerberos principal which you can subsequently use to access all services on the cluster after the migration is done. Your migrated password will be usable with mail (IMAP/POP3) immediately. The migration URL is the following: https://hermes.ugcs.caltech.edu/password.html The SHA1 fingerprint of the temporary self-signed certificate (until we have time to properly establish a CA) is 22:44:7D:F3:D9:44:A0:59:CA:B4:AC:70:5A:F5:94:9A:3F:2C:4F:15

Network Filesystem
We are migrating from NFS to AFS, a filesystem in wide use at other universities including Stanford, MIT, and Carnegie Mellon. AFS has vastly improved security and speed compared to the version of NFS currently in use on the cluster, not to mention better administrative tools which will allow us to easily back up your data and move it between servers to maximize performance. AFS also allows user-settable ACL's, eliminating the need to create custom groups for allowing subsets of users access to data. However, there are a few caveats: AFS does not store permissions by file, only by directory. We are defaulting to have home directories remain readable only by their owners, with a world-readable public subfolder. If you wish to add a public file to your home directory, place it in the public folder and symlink the filename in your private home directory to the equivalent in your public folder. We have already set up a few such commonly-used symlinks on your behalf such as .plan. We will migrate your data for you from NFS and place it in your home directory during the migration.

We have acquired approximately 3.2 terabytes of mass storage and 0.3 terabytes of fast SAS storage. As a consequence, we are setting initial quotas to 500 MB of mass storage for your home directory and 150 MB of fast storage for your mail. We reserve the right to modify these quotas in the future, although they will most likely rise. If you wish to have a larger mail quota, please contact us - we can move your mail spool to one of the mass storage machines and give you more space (at the penalty of performance).

SSH
Your SSH keys will no longer function. This is deliberate - AFS uses Kerberos for authentication, which means that a Kerberos ticket is required to mount your home directory; SSH keys cannot not provide Kerberos authentication. If you SSH to a machine directly and enter your password, Kerberos tickets and AFS tokens are automatically obtained for you using your password. If you wish to use passwordless authentication, we recommend that you install a Kerberos client on your system and enable forwarding of tickets over SSH (GSSAPIAuthentication and GSSAPIDelegateCredentials) for *.ugcs.caltech.edu in your .ssh/config file if using *nix.

We are in the process of acquiring a number of new user-accessible Core 2 Duo systems, but all of the puke-class Pentium III machines will be migrated for the present and the servers used for UGCS 3.0 services will be decommissioned over time and integrated into the cluster as user-accessible shell systems.

Mail
We have switched to using Maildir format for delivery of all new messages. Maildirs perform significantly better in a network filesystem environment by avoiding the need to lock a single mbox file. IMAP and POP will only show messages from your Maildir. We have used mb2md (http://batleth.sapienti-sat.org/projects/mb2md/) to place all the messages from the mboxes we could identify in your Maildir. If you wish to manually migrate additional mboxes after the migration, you can invoke mb2md yourself. All inbound e-mail is now filtered using amavisd, spamassassin, and clamav. If you wish to forward your mail to another address, you should update your LDAP entry with one (or multiple) mailForwardingAddress entries instead of relying on .forward. Procmail is currently not in the mail delivery chain, but will be integrated at a later date if it is still required by a large number of users; we anticipate that the new mail stack will suffice for the majority of users that were using procmail to invoke spamassassin or perform filtering. Additionally, since we are now able to filter all inbound mail, we no longer need to greylist e-mails and therefore you will no longer experience delays in delivery of mail to ugcs addresses. We have disabled non-secure IMAP and POP; you will need to use IMAP/S or POP/S instead. Like SSH, our IMAP and POP services are Kerberized and you can authenticate without entering a password if you have a Ticket-Granting-Ticket. If you wish to send outbound e-mail using UGCS's SMTP server, you also will need to authenticate either using your password or a Kerberos ticket.

Webmail
We are offering two new options for accessing your e-mail from a web browser. Roundcube is an AJAX webmail client that behaves like a desktop mail client with drag and drop support. Squirrelmail is more traditional and works for the more paranoid about Javascript. You can go to https://hermes.ugcs.caltech.edu/roundcube or https://hermes.ugcs.caltech.edu/squirrelmail to access them.

Mailing Lists
We will be migrating all /ug/drop/mail lists to Mailman, a widely used mailing list management tool that offers additional features such as automatic removal of spammy messages, blocking of posts from non-members, moderation, unsubscription, and archiving of messages. Existing /ug/drop/mail lists have been frozen in preparation for the migration. We are offering a web-based list administration tool located at https://hermes.ugcs.caltech.edu in place of /ug/drop/mail. For those who use automated tools to manage /ug/drop/mail lists, please contact us and we will advise you of the best way to handle automatic additions/removals of list members.

Public webhosting
Your public_html folder will be automatically migrated and be served from our new webserver. We support PHP (version 5) and Perl through SuExec. By default, the web server will not be able to read files from your home directory - if your website relies on files outside of the public_html directory, they should by symlinked or moved into ~/public/public_html/. If you have questions about migrating your existing web applications, please contact us.

Software
Cluster machines will be running Debian testing (Lenny) with a set of commonly used packages. If you'd like to request a piece of software which is currently not installed, please contact us and we'll add it to the standard system image. We hope that this central package management will allow us to keep the software on UGCS as up-to-date as possible with new versions and security updates.

Database services
Currently, database migration is not automated. Please contact us to get your database created and/or migrated.

Chat
In addition to continuing to support Gale, we are planning to set up a Jabber server for your chatting convenience.

Hosting and Authentication
As the result of rearrangements made to our very limited pool of 62 usable IP addresses, we have needed to change the block of addresses allocated to third-party hosting. If you are hosting a server with us and we have your contact information, we will send you your new information to place in /etc/network/interfaces and will expect you to configure your server appropriately or provide us with the access to change the IP ourselves. If not, you will have to track us down when your server stops working. In particular, there are a few bits you _need_ to pay attention to with respect to specifying the correct MTU, netmask, and routes. Also, if your server remains offline for more than a period of two weeks and we have no contact information on file for you, we reserve the right to reallocate your IP to someone else. If your server is currently offline, we cannot automatically gather its MAC address and will need this information from you if you wish to have an allocation in the new network scheme.

As always, individuals in the Caltech community are welcome to colocate servers with us. We ask that you provide us with current contact information in event we need to disrupt service to your server; we also require your server's MAC address in order to place it on the appropriate VLAN and provision you with a static IP. We run network intrusion detection software (Snort) to protect your server and also can tighten firewall rules to restrict inbound traffic if you so desire.

Our Kerberos infrastructure is also available to others operating web or other applications who need to validate the identity of a member of the Caltech community. Contact us for details if you are interested.

Regards,

Your UGCS sysadmins (Elizabeth Fong, Matthew Maurer, Joshua Hutchins, and Alex Roper) sysadmins@ugcs.caltech.edu -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG6QL8aj+LAPvd0qQRAuR4AJ0R7g4+EBAreYV9qPENRMTpsIpEhACdHgpn hUKMufALmxRFTL2JpYpZGJk= =Pv2m -END PGP SIGNATURE-

August 22, 2007
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1

Dear UGCS users,

As the 20-year anniversary of UGCS in 2009 approaches, we are making preparations for the next 20 years of UGCS in order to ensure that the cluster is used by as many people as possible and continues to provide top-notch services to the Caltech community. We are proud to announce that we have been planning significant hardware and software upgrades to UGCS over the course of the past six months which will result in vastly improved performance, features, and quotas. In short, we are moving all UGCS services to new, faster hardware and retooling the software architecture to use commodity, well-supported software that we can update and maintain in the coming years.

We hope to be finished with the initial migration by the beginning of October. Please be advised that some UGCS services may need to be temporarily disrupted during the buildout. Additionally, we may snapshot the /ug/drop/mail system and the user password database for migration; any changes following the snapshot will need to be reapplied after the migration. A week before the migration, we will advise you of what changes will impact you and any actions you may need to take. When we switch over to the new infrastructure, we will need to bring down all UGCS services for approximately one day.

If you have any questions, comments, or concerns, please send us an e-mail at sysadmins@ugcs.caltech.edu and we'll respond as soon as we can.

Sincerely,

Your sysadmins (Elizabeth Fong, Matthew Maurer, Joshua Hutchins, and Alex Roper) sysadmins@ugcs.caltech.edu -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGzOZ4aj+LAPvd0qQRAgANAJ0RPhpm5+Lwwk10ItEZdEivgx/UlwCggVnn TaPLKRdEjt2yGcnDH7ygtw4= =qoiU -END PGP SIGNATURE-