bcrc
Rodger is using computers in the intro labs to run a giant phylogenetic analysis for his dissertation. He was planning to walk around with a flash drive to run the analyses, but I persuaded him to let me build some resources to do it over the network. For a long time, I've wanted to have a mechanism that would facilitate people distributing jobs out to all the workstations in our computer labs. We built the first part on Friday.
I started by looking at Apple's "Folder Actions" -- I thought maybe we could just create a folder that we could put stuff in and have an analysis run. But it turns out the Folder Actions are bound to a particular user and only work when that user is logged in. Fail. The mechanism that makes them work is also spread out all over the freaking file system: parts in ~/Library/Preferences and in /private/var/db/ and uses an undocumented faceless binary that is located in some hidden folder someplace. Ugh. So much of what Apple does looks shiny on the outside, but is implemented in a way that's seriously difficult to build on.
After wasting an hour building an Applescript that would implement a folder action, I started over and made a launchd file that would start a PHP script to run the analysis. It works great: it grabs a file, makes a directory with the same name (minus the extension), puts the file inside, and launches Beast. And same strategy could be used to launch any other kind of analysis or process.
We ran into a few bugs getting the whole thing working: first, we found that Beast wasn't getting launched with enough memory, so we had to modify the flags it was getting launched with. Then we had to get it out to all the machines in the intro labs, and we found a few machines that hadn't gotten updated properly and had to fix those. Then we ran into a really baffling problem: we could only ssh into one of the four intro labs.
At first, when Rodger reported it, I didn't believe that the problem was happening. Then I tried it and found that it was in fact true: I could ssh into the one new lab, but not into the other three older labs. I alerted the technical staff and George and I walked through everything: Yes, the machines were configured properly. Yes, you could ssh into them from themselves. Yes, you could ssh into them from other servers. We narrowed down the problem to just the one server: it could ssh into one lab but not the others. We looked at the arp tables, checked the netmask, nothing. At the end of the day, we rebooted the server, but the problem persisted. Then George suggested checking the name resolution. "Right! Maybe it's the hosts file," I said. "It doesn't have a hosts file," George said. "Oh, yes it does," I said. And, in fact, the hosts file was ancient and had bogus entries for the intro lab machines that weren't working. Once we cleaned those out, it worked fine. Whew.
The next step in the system is to set up shared-key authentication for moving the files and an automated system to put the files across, poll for completion of the job, and then pull the results back. But even with that, it's still a lot more convenient than walking around with a flash drive.
When it became clear that we weren't going to get snapper migrated onto mithlond, I went to build course sites on snapper and found that we couldn't do it with php 4 anymore. So I built php 5.2.14 and installed it. It broke some things, but surprisingly little. I've now updated eaccelerator too -- no problems. I ran into some problems getting drush working, but figured them out (the wrong libraries were getting loaded). The course sites are up, in php 6, with ldap authentication, and working in a multi-site install. Check.
I had hoped to update all of the labs to 10.6 this summer. I got the BCRC updated to 10.6, but had enough problems with it that I decided to leave the teaching labs at 10.5. Little by little, I'm getting updates built for the intro labs and, today, spent most of the day today in the ISB getting updates built and applying them. Check.
When I built my 10.6 install, I updated radmind and built a bunch of new transcripts. It turns out that the updated radmind makes transcripts that aren't backward compatible. So I've been carefully making sure I get the transcript for the new radmind installed on all the machines before any of the new transcripts that break the old radmind are installed. Check.
And then there's the whole update-macosx-and-break-X11 that's been the pattern for the past year. Now I need to check if updating X11 broke any of the other X11 apps we use. Sigh...
And, on top of everything, are all the frantic people trying to figure out how to use all the updated stuff. Sigh... I just don't have time to do all the stuff I'm trying to do. It's going to be a rough semester.
Little by little some things are coming together. We've got almost everything authenticating against ldap now. There's one small step we have to make for (nearly) everything to be able to authenticate against ldap. I've had several people thank me for the efforts I've made to bring sanity to how authentication was working in the department. And we're finally using a multi-site drupal install for course sites. It's not quite perfect, but close enough to get through the semester and to perfect next time around. And the department site is taking over the roles of a bunch of disparate sites in the past. Things are getting better. But there's so much new we have to do each semester and so much that's still just cobbled together and undocumented.
But I haven't run into any show stoppers yet. Just two more working days before the semester begins. My last big challenge is to get my writing course set up -- while keeping everything else going. It'll happen.
The semester is coming at us fast. Early in the summer, the plan was to replace the BCRC server. In the end, for various reasons, we didn't quite make that happen. For the past few days, I've been sorting out how to build resources for the fall on the old server. Making good progress.
Yesterday, I finally wrote a new script that builds an instance of drupal for a course site using symlinks from a template and runs the database dump, .htaccess, and settings.php through sed to rewrite the course specific parts. It works perfectly except for setting the name and slogan of the site. Upon reflection, I think I'm going to install drush and then call drush after the site is built to set those.
As if I needed another project to work on, I decided to involve myself when the department wanted to have digital signage. I've been skeptical about buying one of the closed-source solutions, since I've heard nightmarish stories about trying to support them on other parts of the campus. They wanted to show a video on one in the ISB and several faculty spent hours trying to make it work before they got me, with ffmpeg to begin systematically exploring the fileformat/codec space until we found one that would work. Stupid. We bought an inexpensive LG display and a MacMini and are planning to use this recipe. Basically, you create a unique URL at your drupal site that presents content with a special theme optimized for your display.
The recipe is really cool. I showed it to Tom to see if he would be interested in working on the special theme. He was super excited because the Dean's office has been interested in solving the digital signage problems they already have.
I've run into some roadblocks. Apple just rev'ed macmini line and so the hardware architecture is completely new. I tried using our existing macosx install and it wouldn't work. I explored putting ubuntu on the macmini and found that the macmini hasn't been out long enough for ubuntu to work on it either: I couldn't even get the installer to boot. I expect it will be fixed in a few weeks, but that was frustrating. So I went back to macosx and built a unique radmind image to support display machines. All the machine has to do is run either plainview or firefox in kiosk mode.
I also ran into a bit of skepticism from the chairman who wondered why we couldn't just use Powerpoint. Sigh... He's concerned about getting faculty to contribute and review content. I pointed out that the person who's probably going to enter most of the content is already putting up the announcements on the website and this will just become an extension of that. If they had to pull all the information out of powerpoint slides to make announcements, it would be more than twice as much work.
I've got my Esperanto course basically ready to go. It's always a challenge to guess how much time outside of class students will be willing to invest. I've aimed at 2 hours. I've outlined 4 tasks to do each week that I estimate will each take around a half hour: 2 lessons at lernu and two chapters from Bonvenon en nia mondo. I think if they actually do everything, by the end they should be pretty fluent in Esperanto -- I guess we'll see.
The writing class I've taught often enough that I'm not worried about it. I still haven't come up with a good theme for the writing class, though. I usually try to come up with some kind of unifying theme that ties all the activities together: invasive species or something. I've been thinking about doing this one about models. When we have to do our final research project in late November, it will be warmer to study a model, rather than try to collect data in the field. I also need to think up a good object for them to investigate during the first day...
I spent an intense week, culminating last night, preparing for and then migrating the department services from an old server to a new one. It's a server we've had for about 10 years and it has everything running on it. I was primarily responsible for migrating the file and web services -- including a complex scientific web application. Last night, I was in the office until 3am trying to stitch things back together. Today, most things are sort of working, but many things aren't quite right. Still -- we did it and the world did not end. Now we can go forward and make things better.
Today, we went to see Jonathan Coulton perform with his new band. He played a mix of old and new songs. The Iron Horse was packed with his fans, who sang along with all the old favorites and listened raptly each time he performed something new. We always enjoy seeing him perform, although its quite different when he doesn't have Paul and Storm to warm up the audience. On stage, he affects having quite an edge, which works for him pretty well.
I had seen that John Hodgman was going to be there and when he saw that I was going to be there he mentioned via twitter (I guess that's called "tweeting it" to use the vernacular.) Within minutes, I had another half-dozen followers on twitter. I wonder how long before they discover that, instead of some interesting funny man, they're following some weirdo who writes Esperanto-language haiku.
Afterwards, I posted two comments, I said
limako was not just soft-rocked by @jonathancoulton & actually touched @hodgman with (his) hand / devas nun gardi la manon kontraŭ memorigaĵ-stelistoj
and
famulo staras inter la homar'... kvazaŭ mem homo / the celebrity stands among the people... as if one himself. #hajko #haiku #senrujo
It was nice to meet John Hodgman. I shook his hand and he spoke with me for a few moments. He was very genuine and thanked me effusively for the little translation I made for him several months ago. He invited me to speak with him further after the concert, but he was swarmed with people, so I just remembered myself to him in passing as I left, with his earnest thanks following me. I was pleased that Lucy got a copy of the book that had that bit for Richard for his birthday.
After we got back, we finished packing for St. Croix. Tomorrow we leave early for our next big adventure.
When I was hired at UMass, I was largely given carte blanche to do whatever I thought needed to be done to improve how technology and education were implemented in the department. It was very uncomfortable for the first couple of years to be so unaccountable. I kept waiting for the other shoe to drop: for someone to tell me I wasn't doing the right things. I did have a steering committee that I could go to for direction, but they ended up seeming more like a "pep squad" than something to be accountable to. My goal has always been to maximize the utility of technology for users: to reduce obstacles and make it easier for people to do what they need to do.
Unfortunately, over the last couple of years, the pressure has been to demonstrate accountability and reduce costs. So, for the past year, I've been putting in place infrastructure that makes the facilities harder to use: you can't just log in and print anymore. Now people have to jump through a series of hoops to see if they have an account and then more hoops every time they want to print. By implementing page quotas on printing and requiring an account to log in, we get much more information about who is using the facility -- and it does provide better service in the sense that we can exclude the freeloading non-biology students that didn't really belong there in the first place. There are fewer lines now and much less wasted printing: people don't print unless they're pretty sure they want it.
There are just two or three more technical hurdles to overcome and almost our whole infrastructure will be able to use the authentication system we've been building. I think it's going to make a huge difference in efficiency in setting up and providing services. But it's a big distraction from the kind of development that I'd really rather being doing.
Toward that end, I'm getting an iPod Touch to start thinking about building cool new educational apps. The Apple guy had me look at the "cool new educational apps" he liked, which were just the typical transmissionist bullshit: transmit some "facts" about a topic and give people a quiz to see if they "got it". Or watch a lecture on your iPod! Sigh... We've had this conversation enough times that he knows I don't like that kind of thing, but he doesn't really get why.
I think the first thing I'll build is my simplified take on the "knowledge broker" idea the physics guys had a couple of years ago. They wanted to be able to ask open-ended questions about reasoning in large classes and have some intelligent system to winnow down the answers into something manageable. I don't want an intelligent system -- I want the intelligence to be in people. I had the idea of letting people enter answers, but also to see a ranked list of answers submitted by other people and to vote for them. Most people would probably not enter anything, but would just vote. In a large class, you'd see enough answers to have an interesting sense of the diversity and the top 5 or 6 would probably capture the sense of the class. (Maybe only let people start voting after you have 5 answers?) Furthermore, you could leave the voting open while you discussed the question and, as various options got eliminated, you could eventually come to consensus on the best answer. I think I could build something like that based on Duck given a long weekend. But it would be coolest to have it on some portable device, like an iphone or ipod touch. Once I get the device, I'll start looking in to building it. It should be fun.
Since I arrived at UMass, we haven't had a good mechanism to keep track of who's using the BCRC. There was a practice of having people sign-in on paper sheets, which we continued on inertia, but it was never very satisfactory. People just scribbled whatever. Sometimes sheets weren't available. The sheets got lost. And, in the end, it was not a very useful form for data. At the end of the year, I would sample the data and do an analysis to estimate usage in the BCRC.
I had pushed for years (practically since I arrived) to have student accounts that we could use to have students log into the workstations. But with the technical staff stretched as thin as they are, I could never get the issue high enough on the list of priorities to make it happen. When we began working in the ISB, I finally prevailed on George to let me just build the infrastructure we needed. We did pilot-testing last spring and this spring, we're actually doing the deployment. There's still a lot of work to be done, but finally, we're building accounts in a form that we can use them to have students log into our workstations. The generic login is still available, but soon everyone will be required to use a real account to login.
What we still didn't have was a mechanism to record the login data somehow. I had been thinking about the problem for a while and decided that the easiest thing would be to have each computer use curl to talk to a webpage on the server with the information: all we need is the username and the name of the workstation. That gets all the information to the server, but then how to record it. Writing to a flat file wouldn't work because you'd have multiple apache processes all trying to write the file at the same time. Using a database would be stupid because all you need is to write a stupid log file. Then I remembered that PHP offers an api for talking to syslog!
We set up a log local channel in syslog and I wrote a tiny script that just accepts the data and passes it to syslog. Now we have a log file that shows me each time someone logs in. No more sign-in sheets! At the end of the semester, we should be able to develop much more robust data about who's using the BCRC and all of our computer labs.
I've written a print release system that we're using to manage printing in Biology. We have to run the print release system configured differently between wahoo (in the ISB) and snapper (in Morrill). On snapper it doesn't have Drupal as its parent and on wahoo, we have to have it manage queues on wahoo, but use the database on snapper to maintain page quotas, etc. Trying to keep a common codebase between the two installations is problematic. I'm trying to keep the configuration info separate from the code, but it's not perfect.
Since I last updated the print release system on wahoo, we'd been having odd problems. Occasionally, when you'd load one of the print release system pages, you'd get a weird error out of Drupal in the parent directory. (The error looked like drupal was trying to run as if it was inside the print release folder and it couldn't find its associated files to load). There didn't seem to be any correlation between which action you took and the appearance of the error. I did a lot of experimentation and eventually tracked it down (I think).
In the .htaccess file in the drupal directory, Drupal sets itself up to be the agent responsible for 404 errors. In the css file for the print release system, I was trying to load the background graphic from snapper (a picture of a snapper) instead of for wahoo (a picture of a wahoo). That graphic didn't exist on wahoo, and so it caused a 404 error. And the 404 error would prompt apache to try to invoke Drupal. And Drupal couldn't run in that context. Or something. I still can't figure out why it would only break the thing randomly. But it was a very weird manifestation of a simple, boneheaded problem which I think I have corrected.
One of the challenges that George and I confronted soon after I arrived at UMass was how to provision computer services to students. At the time, the Biology Department had two servers -- a Sparc 10 and a Sparc 20, I think -- and all accounts were built at the comment line manually by George once on each server. The Intro Biology course was among the first that wanted to have accounts and, with a population of 700 - 800 it was going to require some automation.
A brief aside: when I described what we were doing to Jack Wilson, who was then the director of UMass Online, before he became president, he asked why accounts weren't being generated centrally and I explained that there were no central accounts that students could be depended on to have because the fees to support the technology services were optional and some significant fraction of students chose not to pay them. That meant that, as an instructor, if you wanted to be certain students would have a particular resource, you needed to build it yourself. We also didn't have any means of associating student usernames with individuals nor any standard way for us to check students' authentication either. But I digress.
George and I discussed what to do and drew up a plan to synchronize accounts between the two servers and to build student accounts only on the BCRC server. In order to avoid collisions with existing usernames, we added a digit based on the year of enrollment to the end of a username generated by munging first initial and last name together: the system would start out with sensible versions and then try various other permutations until it found one that was unique. Once a username was assigned to a student, they would get the same username from then on. George could then script building accounts and I mostly used imap to let students authenticate against the system.
A key challenge was resetting passwords. We chose to set students' initial passwords to their student number. I wrote a password changing script that used poppassd and we encouraged students to change their password. If a student forgot their password, I would reset it based on email from their @student account or if they visited my office with ID. The whole system was not perfect, but it worked reasonably well.
When we started using smb authentication for shared file space and printing, we had to start setting smbpasswords at the same time as unix passwords. Then, we added a new server in the ISB that needed to have some accounts replicated. Then we started building replacement servers for the Department and BCRC. At that point, I finally prevailed on George that we needed to have a central authentication system to bring some sanity to passwords. It was a constant problem for people to try to figure out which system they were authenticating against to debug problems: "Let's see... This is your smb on wahoo which should be the same as your SMB password on marlin, which might be the same as your email password, but might be different."
Last spring I set up ldap for the first time and we ran a pilot project in the ISB. We built all our usual accounts, but set samba on wahoo to authenticate against ldap. The original plan had been to merge our account generation stuff for undergraduates with the similar efforts in Chemistry. We share so many students and have shared resources in the ISB that it makes sense to merge how we provide authentication. Their system wasn't ready in time to use for last Spring, so we ran our pilot project. Then we were supposed to merge our systems over the summer, but we found at the last minute that we couldn't use their set up (because they hadn't built ldap with crypt or turned on the apple ldap schema we were using), so we postponed and reused the pilot work from the spring. I had hoped to make the merge happen over intersession, but there was another fly in the ointment.
While we had been using the student number as a password, Chemistry had been using it as a username. During security discussions with OIT, they indicated that we probably shouldn't be using the student number at all -- but especially not as the username. OIT does now provide accounts to all students now and furthermore they were willing to give us rosters with student usernames and to open a means for us to check student authentication for the purpose of setting the same username with either the same or different password at the student's choice for our use.
Over break, I set up a new authentication server for ongoing ldap services and build a password setting script that let's students authenticate using pubcookie and sets our ldap password. I had hoped to get the structure consistent with Chemistry but we couldn't get a response from them in time, so we just went forward with the same structure we'd been using for our pilot project. But we also migrated all of our permanent accounts into the system, rebuilt the account generation scripts to use the new roster format (which includes the NetID), modified everything to point at the new ldap server, and built 3750 student accouts. So far, the system is providing authentication for samba file service and printing on two servers, our print-release system, and duck.
There's still more to do: we still have to get shell service on the servers to use the authentication system, and I'm planning that we'll also use it for our course websites and, little by little, pretty much everything else. If we can ever get on the same page with Chemistry, we'll have to go back and tweak everything to use a different DN. Hopefully, we can do that over the summer. But we've made substantial progress in a very short time.
The semester has started and I'm feeling busier than I think I've ever felt. I'm being pressed to support more, more complex, and more dispersed resources. Every way I can think to describe how busy I am feels cliched.
I've introduced a big change in the BCRC: I've set limits on free printing. This summer, I had a volunteer who worked out with me the general outline of a web-based page release system that could be grafted in between Samba and lp (or any other unix printing system). We set the print command in samba to calculate the number of pages in the job and submit with it job-handling set to hold. After authenticating, the user can see their jobs and page totals, they can select jobs, and then they can release (or cancel) them. The system has a few tables in mysql to keep track of jobs and page quotas. We're still feeling our way forward toward a system for establishing personal page quotas and providing exceptions.
I didn't get all 40 hours of work done, but with intense efforts on Saturday, Sunday, and Monday, I got the machines needed for Tuesday set up and some more besides. I still have a bunch more work to do, but nothing critical.
On Friday, we got the network minimally functional and I set up an assembly line to update laptops. On Saturday, I had three drives that I could use to wipe a laptop and install a stripped down image. Reboot, launch the radmind script and, in about half an hour, the laptop was ready to go. In between, I played some SC with Phil and watched episodes of Natsume Yujincho. I find them charming and heartwarming. I got most of the laptops done.
On Sunday, I came in to finish up the laptops and work on the Kodak Molecular Imaging software. I built a transcript of the package and found that the ownerships were wrong on the server, so I was stuck. I sent a note to the technical staff and used the excuse to take off to see Coraline in 3D with Charlie and Lucy.
On Monday, I opened up the BCRC, made sure things were still working in Morrill, and headed over to the ISB. In between working on the lab, I met with a visiting faculty member from Ohio State who is here observing our use of instructional technologies. I helped him get to Zane's genetics class and then spoke with him for an hour or two during the afternoon. Late in the day, Kate and then George arrived and we finished getting the room set up for class tomorrow.
I will be out of town tomorrow, going on an excursion to UConn to meet with the collections director. So I won't be there if things collapse in the new classroom. But I think things are going to be just fine.
Since the spring, the relations between the UMass Amherst faculty, president, and board of trustees has been strained. When the trustees announced plans to have a retreat with the goal of generating action steps, there was some concern raised. Today we held a general meeting of the faculty with a set of resolutions in place to express our lack of confidence. But the trustees decided to have an open meeting, rather than meeting in a secret retreat and so we backed away from the resolutions. One trustee, James Karram, attended the meeting and spoke briefly. I'm encouraged by the positive steps and I hope we are beginning to move away from the series of confrontations that have driven faculty governance relations since the spring.
I was involved in drafting a resolution for the meeting related to the governor's Readiness Project -- this is a commission that has been drawn up to study education in the state. One subcommittee is supposed to study public higher education. Pat Crosson, a highly respected former-provost, is on the committee, but no current faculty, staff, students. We drafted a separate resolution to encourage the commission to hold open forums at the UMass campuses. I drafted the following resolution (with help from many others) to try to provide some guidance to the new readiness project subcommittee:
The Faculty and Librarians of the University of Massachusetts Amherst appreciate the Governor
For the past two years, I've been mentoring a group of Charlie's friends that are interested in computer stuff. We set up Muppyville and have continued to interact about programming. A few months ago, a mother of one of them asked me if I was aware of any opportunities for learning computer programming. I had been thinking about starting a computer programming club, but hadn't quite willing to commit myself. But I decided it was time. So I checked with the department, to make sure it was OK to bring kids into the BCRC and got the parent of another child (who I was sure would be interested) to come with me (mainly so I wouldn't be alone with the whole bunch of kids). Today was our first meeting.
Recent comments
13 weeks 1 day ago
14 weeks 1 day ago
27 weeks 3 days ago
34 weeks 6 days ago
38 weeks 3 days ago
38 weeks 3 days ago
44 weeks 7 hours ago
45 weeks 1 day ago
45 weeks 5 days ago
45 weeks 6 days ago