We updated the BCRC server over spring break. We've known we needed to do it for a long time now, but have been putting it off because we have old Drupal 6 sites that need to be migrated, I've believed that they (or some of their modules) will probably not work with newer version of PHP in this version of Ubuntu. During the Fall, we updated the server in the ISB and, to prepare, we tested and I rewrote some of the old legacy code that I had written. So we knew that much. But this server was enormously more complicated.
It turned out there were only two serious problems with my old code. Originally, PHP used its own regular expressions library (ereg) but, at some point, started including functions against the perl regular-expressions library (preg). I've probably been using PHP since before it had either. But I had to make minor API changes and update my functions to use perl regular-expressions. I also had start using mysql very early and those functions had become deprecated, so I needed to make minor changes to all of those.
The first serious problem was that our MySQL databases didn't get updated properly. The updater is supposed to be able to ask mysql to update the databases, but something went wrong. At first, mysql wouldn't start at all, I think because it was confused about which my.cnf it should be reading. This was probably why the update didn't work correctly. This server has existed since ~1995 and was originally a Sparc 10. Then we got an E250, then a T5220, and most recently a Supermicro running Ubuntu. But a lot -- too much -- of that history is still there. Once I cleaned up the my.cnf files, we got mysql running, but it couldn't read the database of users and so nothing could get access to its data. It turns out that to update the database, you need to log in as "root" so if you can't log in, you can update the database. So then I shut down the service and started it manually with --skip-grant-tables. Then I was able to run the upgrade script. Then mysql came up. (Note this story actually skips several epicycles I made trying to sort out the problems, but is what I did in the end.)
Before the migration, I had started re-building a new web-tree around Drupal 8 (rather the old tree which was built around Drupal 6) hosted at a different CNAME. I had migrated the key pieces of what I wanted going forward, without taking down the old web-tree. After the update, I switched to the new web-tree and then migrate stuff out of the old tree to the new one. I figured this was potentially more disruptive, but safer: there was a lot of old history that was infested with bitrot. It seemed better to start with known good things and then rescue stuff that people identified as missing after-the-fact.
There are still a lot of rough edges: broken paths, missing images, stuff that needs to be reconfigured. But basically good to go. The two most complicated updates are yet to come. But this was a good test-bed to help us prepare for what we need to do for those.