I really have to stop listening to the same song played over and over. It may affect my thinking….

We had another node crash due to the Sun JVM issue. Our start script failed to make a file in /var so the node did not become fully operational as expected. While waiting for those with permission to delete some stuff to free up space, I went looking for what I could delete myself. Naturally /var/tmp seemed a likely place. I found 1,171 files named Axis#####axis. (Replace the #s with well… numbers.) They used up only 42MB. Most were small. Looking across all our machines there are thousands of these dating back to February of this year.

I love the Unix file command. It will tell you what kind of files are there. So I used file | sort -k 2 to sort by the type. Almost all of the files were either plain text or JPEG or GIFs. One file, called a “c program file” turned out to be a JavaScript (based on the C syntax). I downloaded a JPEG file locally, renamed it to have the .jpg extension, and opened it in an image viewer. It opened correctly. Seems its a graphic of a table.

It would seem our Blackboard Vista 3 has been collecting these files for months. They do not take up very much space. There are not nearly enough files to represent a download of content by all users. Our /var would fill up hourly in that case.

Axis is an Apache SOAP project. Vista’s exposed APIs use Axis, I believe. So, the running hypothesis is several of our campuses are using a product which is contacting the APIs to upload content. Its spread out enough that all four clusters are affected. Its something that started about February.

Suspect #1 Respondus – Chosen because we know it hits the APIs to upload content. Discounted because the content is lecture materials. Respondus works with assessments (aka quizzes, tests, exams).

Suspect #2 Impatica – Chosen because the JavaScript file references PPT. Impatica compacts PowerPoint (aka PPT) files and allows them to play without needing a PPT player. Their support pages teach users how to use the Campus Edition 4 user interface to upload content into a course. O-kay….

Suspects #n Softchalk, Diploma, Microsoft .Learn, etc. – I haven’t really investigated any of these. They are just names to me at the moment.

UPDATE: So… There is a bug in Axis which dumps these files into the file system. The files can be deleted as long as they are not current.