You are viewing a read-only archive of the Blogs.Harvard network. Learn more.

Finishing Touches on the Interface

ø

Gave myself a little bit of a rest this morning, as a reward for successes yesterday—took about an hour to leisurely give the houseplants a bath, as they had accumulated a thick layer of dust and were looking very dry and sad. Then turned my attention to fixing the remaining bits missing from the interface, one of which is to record the objective/magnification of the images taken. I thought I’d be able to do this by adding or modifying an entry to the JPEG EXIF metadata, but once I’d found the appropriate function I spent quite a bit of time trying to do this and not understanding why it wouldn’t work. It was only when I read the manual (I know, I know… if all else fails!) that I realized that—bizarrelyImageJ only supports saving metadata for TIFF and ZIP file formats, but not for JPGs. Very odd, but at least it explains what was going on. Saving the files as TIFFs isn’t a great option, as the file size balloons from ~600KB per file to ~30MB. I’m reluctant to add a field to the SQL database to contain this info, because that would mean tweaking both the database set-up, and several functions within the R program that handle formulating the SQL INSERT statements. Not that it’d be a lot of work—but the thing works and I don’t want to risk introducing bugs.

Aha! What about appending the objective label to the filename? The filename’s already being stored in RadData, so this would be a super low-impact way of associating the data with the file directly.

Mike Foote Visits

ø

The last three days have been a whirlwind of busyness with rather little in the way of sleep. While there was some fun socializing involved (dinner with Tinker and friends on Friday, a visit with Pierre on Saturday, and brunch at Evan and Katie’s on Sunday), there was a lot of intense wedding work (the website all of Sunday afternoon and evening, the invitations Thursday evening past midnight and Friday from early morning through four in the afternoon), and no work at all preparing for the meeting with Mike Foote today.

This lack of preparation has left me something of a gibbering wreck of nervous, and resulted in another low-sleep night last night. (The night before it was the cat, who decided to wake up and mew incessantly from 3:40 am until I finally got up at about 6 after trying time and again to get back to sleep. She promptly curled up and fell asleep on the bed.) All manner of destructive thoughts of the familiar variety—regret and shame over the many years with no tangible results to show, no product to talk about—came back to haunt me as the hours ticked on. Not helpful to think about, of course, but in mapping out how the conversation might go, it was difficult to avoid the difficult questions that might very easily come up—so, what have you done over the past five years? Why didn’t those projects work out? Not things I much enjoy thinking or talking about.

It certainly also didn’t help that the decision to meet with Mike, while ultimately made by me, was urged on externally. For much of the last few days I couldn’t quite put my finger on why exactly I was meeting with him—Jerry’s justification was that he thought I should meet with him so I could scope out the possibility of doing a post-doc in Chicago, but I’m neither interested in a post-doc, nor moving to Chicago (feelings difficult to comprehend for an academic, of course), so this left me somewhat bereft of a purpose for our conversation. Of course, if I had something more tangible to talk about—for example, a first plot of the morphospace results or some measurements of radiolarians—I might well be able to ask his advice in how to analyze or interpret this data. But given that I don’t have any results yet (there again, the deep-rooted source of my anxiety), there’s not much to talk about. I can (and probably will) talk to him about what I’m planning to do, and how I’m doing it—the morphospace character choices, the link between the morphospace and the Neptune database, the Fourier-based outline morphospace, the database and measurement interface for the radiolarian lineage project—but it’s not quite clear in those conversations what I’d actually be asking him—other than to sit and listen to what I’m doing. Kati made the very helpful point in the morning that I was meeting a human (of sorts—at least in all likelihood), and it was OK to meet someone just to get to know them, because you have overlapping interests, and because you work on similar things. It’s OK to meet someone because you although you don’t need or want anything from them, like a job, you’d like to know them because they might know somebody who might want to give you a job. I don’t want to post-doc with Mike Foote in Chicago, but he might know somebody at the small liberal arts school on the East coast that wants to give me a job.

The chat didn’t go as badly as I had feared. If anything, I talked too much—I had been so nervous about not having enough to talk about that I had gone over and over in my head what I could talk about to fill the deadly silence I feared, that I might have steamrollered Mike a little bit. I think I would have bored him to death with my monologue had I not had the foresight to suggest we go for a walk outside—it being a reasonably warm day and he, presumably, having spent the morning locked up in stuffy offices. This turned out to be a great suggestion, and one he seemed very keen to follow. We walked around and I rabbited on about my projects, and asked him about his opinions on the new Alroy subsampling algorithm. The only point where I might have blown it, or done a particularly good job (I’m just not sure), was when he raised the question of the future at the end: “So, what are you going to be doing next? Looking for a post-doc,… etc?” which may have been out of politeness and curiosity only or may have been some sort of exploration toward my availability. Either way, I answered perhaps a bit too candidly, and gave him my honest feelings about not being sure whether the academic path was for me, how hard I had found it to deal with so much effort going into so many failed or stalled projects, but that I really enjoyed teaching and wasn’t ruling out academia. I expected this to elicit a totally dismissive response, or a “ah, you’re one of those, I’m not interested in talking to you anymore” glazing-over of the eyes, but he actually had something nice to say. While I don’t think he had anything like the sort of PhD experience I did (he finished in four years, if I correctly overheard the end of an anecdote he was telling the student he was with when I went to pick him up), he said that he’d had a number of projects that had failed over the years, and one in particular that he’d keep coming back to every year or so for a couple of months, get frustrated with, and then drop. In the end he said that particular one was the project he’s most proud of in his career, because in spite of having started it in about 1995 he eventually did recently crack it and publish it, and he told of how the reward did pay off much of the frustration. He also said, which I felt was a very supportive thing to say, that if everything you work on pans out, then you’re working on the wrong things.

Anyhow, above all else I was delighted that it was over, and between that relief and a working database interface, I was happy to call it a good day.

Life Just Got Busy

ø

Since DSA yesterday afternoon. Once off the phone with Beau, I rushed home to get Kati ready for her flight to DC. Spent a much longer evening than usual working on the invitations—glue, ribbons, envelopes, calligraphy, etc—and didn’t get to bed until well after midnight. This morning, thought I’d be able to finish the rest in a quick whirl before work, but it ended up swallowing up most of the day and it was 3:30 by the time I finally made it to work via the post office to drop off the 72 beauties in their handwritten, bright yellow envelopes.

In the midst of the mounting stress over the upcoming meeting with Mike Foote, and the PlanktonTech report I have to write,  and the status meeting with Andy to prepare for, and the résumé that needs putting together, and the R interface to finish up, I got an email from Dave Lazarus asking for comments on a paper he and John Barron are planning to submit to Nature. On—wait for it!—Cenozoic diatom diversity. Yes, believe it or not, I just got scooped once again. On work that I haven’t done yet, so I guess it’s not technically being scooped if it’s something you were thinking about doing someday but hadn’t gotten around to. Anyway, this is another thing on my plate and while I’m sure it’ll be fascinating to read it’s also another blow to my confidence for not getting that project done quickly, not to mention several hours next week down the drain at a time when I really need them. But, I also need Dave, so there’s no way I can blow him off—though I did email him back to ask how much feedback he wanted, and that I wouldn’t be able to provide much before late next week, because I’m busy. Humph.

Anyway. Now that I’ve vented some time-crunch stress, I’ll get to the task I started (but didn’t finish) yesterday—putting the keystone pieces into the Bridge of Rads database interface. Once in place, we’ll see whether I can jump up and down on it without it crumbling into a sad pile of rubble, as the dust swirls around me.

Made some decent progress over the course of the afternoon, putting all the major pieces in place—but noticed that there were quite a lot of little pieces missing altogether. This is the downside of “sit down and code” rather than developing a very detailed plan of what the program’s going to look like… But whatever, it’s not an enormously complex piece of software, and I can figure out as I go along.

Left work at 7:15pm, by which point I’d gotten much of the way through—although I have yet to do a first test run of the whole interface to see if it all works… perhaps tomorrow?!

Tying the Interface Together

ø

Completely neglected to post yesterday—it was a slow, distracted day but I still got a few things done—mostly assembling bits and pieces of the R routine to read the ImageJ “pipe” file into R and create the SQL INSERT statements.

In the course of thinking that through, made a minor design change in the database. Rather than storing the complete name of each measurement type in the relevant column of the “Measurements” table, e.g. “Length from top of shell to base of ante-/postcephalic chamber”, I thought it might be prudent to store just an integer ID related 1-to-1 to a measurement name. Wrote a quick look-up function to return the full name for each integer ID so that the original design can be easily recovered with a function call. The rationale was that once I have finished collecting data and am at the data analysis stage, it might be very clumsy to formulate the mathematical models in terms of SQL queries including those long labels, and that the volume of a sphere, for example, would be more neatly described as “π x (1)^2” than “π x ‘Width of outer medullary shell’^2”. This might be daft and overkill, but I had a horror vision of running into problems with that and decided to do it this way instead. I think, given that SQLite is so forgiving in terms of data types, it would also be quite easy to revert to my original design and replace the measurement type IDs with the full strings quickly.

Anyhow, that was yesterday, and today is today. For the morning I’d like to get the bits and pieces together and get the interface up and running.

Macros Done!

ø

Well. Another somewhat slow day, but not without progress. Managed to hash out the remainder of the ImageJ portion of the measurement interface. It seems to be quite reliably making measurements, with the intervening calculations of area, and writing them successfully to the “pipe” file—all the while keeping track of the measurement number to facilitate easy reference to the image displayed by the R portion of the interface. Huzzah!

Here the product of the last two days’ work—embarrassing though it may be that it took so long, I’m proud that it seems to work. Many may have been able to do it faster, but far more would not have been able to do it at all…

//Macros to collect radiolarian morphometric measurements for acquisition
//to the RadData database, via its R interface.
//Written by Ben Kotrc, Mar 1st, 2011
//Declare global variables
//Measurement number (in reference to R interface image)
var counter = 0;
//Microscope magnification (e.g. 16x, 40x, etc)
var objective = “??x.”;
//Macro to write all measurements of type “Length” taken since the last
//file write operation, with the current image file name, to the pipe file
macro “Save linear measurements to file [1]” {
//Get name of currently open image file
curfname=getInfo(“image.filename”);
//Loop through each row in the Results table
for (i=counter; i<nResults; i++){
//Extract the ith row from the “Length” column of the Results table
measurement = getResult(“Length”, i);
//Concatenate row of data to be appended to pipe file (data + filename)
insertion = d2s(measurement,2) + “\t” + curfname;
//Append the ith measurement to the Results table
File.append(insertion, “/Users/Ben/Dropbox/Harvard/By-Lineage\ Rads/RadData\ Database/pipefilename.txt”);
//Increment the measurement counter
counter++;
}
//Display in log window how many measurements have been written to file
print(“\\Clear”);
print(“Scale set to ” + objective + “.”);
print(d2s(counter,0) + ” measurements recorded to file.”);
}
//Macro to calculate the pore area proportion from the measurements added
//to the results table since the last file write operation, then write the
//number to file with the current image file name
macro “Save area measurement to file [2]” {
//Get name of currently open image file
curfname=getInfo(“image.filename”);
//Find the sum of all but the first row in the Results table
sum=0;
for (i=counter+1; i<nResults; i++){
sum = sum + getResult(“Area”,i);
}
//Find the pore area proportion
measurement = sum/getResult(“Area”,counter);
//Concatenate row of data to be appended to pipe file (data + filename)
insertion = d2s(measurement,4) + “\t” + curfname;
//Append the result to file
File.append(insertion, “/Users/Ben/Dropbox/Harvard/By-Lineage\ Rads/RadData\ Database/pipefilename.txt”);
//Increment the measurement counter
counter++;
//Display in log window how many measurements have been written to file
print(“\\Clear”);
print(“Scale set to ” + objective + “.”);
print(d2s(counter,0) + ” measurements recorded to file.”);
//Tidy up the Results table
IJ.deleteRows(counter,nResults-1);
setResult(“Area”,counter-1,measurement);
}
//Macro to insert a zero-valued measurement to the pipe file, and update
//the ImageJ results table accordingly
macro “Add zero measurement to file [3]” {
measurement = 0;
curfname = “No file”;
//Concatenate row of data to be appended to pipe file (data + filename)
insertion = d2s(measurement,4) + “\t” + curfname;
//Append the result to file
File.append(insertion, “/Users/Ben/Dropbox/Harvard/By-Lineage\ Rads/RadData\ Database/pipefilename.txt”);
//Update Results table to reflect added zero “measurement”
setResult(“Area”, counter, 0);
//Increment the measurement counter
counter++;
//Display in log window how many measurements have been written to file
print(“\\Clear”);
print(“Scale set to ” + objective + “.”);
print(d2s(counter,0) + ” measurements recorded to file.”);
}
//Macro to reset the tally of the number of measurements written to file,
//as displayed in the log window, to zero (use each time a new individual
//is started)
macro “Reset measurement counter [4]” {
//Clear the Results table
selectWindow(“Results”);
run(“Close”);
//Reset counter
counter = 0;
//Display in log window
print(“\\Clear”);
print(“Scale set to ” + objective + “.”);
print(d2s(counter,0) + ” measurements recorded to file.”);
}
\\Macro to set the appropriate scale for different microscope objectives
macro “Set magnification… [5]” {
Dialog.create(“Set magnification”);
Dialog.addChoice(“Objective:”, newArray(“16x”, “40x”, “80x”, “100x”));
Dialog.show();
objective = Dialog.getChoice();
if (objective == “16x”) {
run(“Set Scale…”, “distance=1190 known=26.3 pixel=1 unit=µm”);
} else if (objective == “40x”){
run(“Set Scale…”, “distance=1190 known=26.3 pixel=1 unit=µm”);
} else if (objective == “80x”){
run(“Set Scale…”, “distance=1190 known=26.3 pixel=1 unit=µm”);
} else if (objective == “1000x”){
run(“Set Scale…”, “distance=1190 known=26.3 pixel=1 unit=µm”);
}
//Print new magnification in log window, along with # of measurements taken
print(“\\Clear”);
print(“Scale set to ” + objective + “.”);
print(d2s(counter,0) + ” measurements recorded to file.”);
}
Well, that was fun, but now it’s time to move on and figure out how to parse the measurements this script writes to file into R and, in swift succession, into SQL queries that can shuttle the measurements safely into the database.

Micro Macro Steps

ø

Started the day finishing Friday’s post—it was another busy weekend and I didn’t get around to finishing off writing up notes from the job fair. It seems that it would be wise to put together some sort of résumé sooner rather than later… It will be nice to be have something to hand out when I meet interesting and relevant people, rather than having to take their card and email them. Although that’s probably not a bad strategy to follow additionally.

Had a slow day the rest of the day, but managed to tackle a couple of the ImageJ macros I needed to make—writing linear measurements to file, and writing area measurements to file. They seem to work nicely. Now I just need macros for setting the magnification, and some way of keeping the counter going that shows which measurement number I’m on.

Shuttling Shit Between ImageJ and R

ø

Incredibly slow moving day. Monkeyed around with getting measurements written to file from ImageJ, which you’d think would be easy. And it is. Don’t know why it’s taking me this long; lack of focus, I guess. Got part of the way through doing this and figured out that I’ll need just three ImageJ macros, plus some counting trickery, to do it all:

  1. Set magnification—a macro or set of macros to set the pixel:µm scale for each of the objectives I’ll be using to make my measurements, the first step after taking a picture down the scope.
  2. Write linear results to file—a macro to extract the values from the “Length” column of the results table, pair them with the image filename, and append them to the pipe file. This will do for all of the length, width, and shell thickness measurements.
  3. Write area results to file—a macro to subtract the 2nd through nth values from the “Area” column of the results table from the 1st such value, then append that value to the pipe file along with the image filename. This will calculate the pore area percentage value from the raw measurements of total area and pore areas.
  4. Number trickery—haven’t figured out exactly how this will work, but since I’ll be making measurements on multiple images with separate results windows, I’ll need to keep track of which measurements I’ve made and which number I’m on. Haven’t figured this part out yet.

Aborted day fairly early due to DSA followed by (yet another) job talk. This the penultimate one, thank goodness.

Radiolarian Catwalk: Here Come the Models

ø

Staggeringly enough, I managed to get the stoopid OmniGrafflery done this morning—in spite of a lateish start (phone call to Anita before breakfast to help organize her travel to the wedding, then dropping off keys to Mark) and with time to spare before lunch. Whoop!

Here are the bitmap files outlining the measurements needed for each species, which will appear in the graphics window of the R interface at the appropriate times:

Now that this task is completed (great big checkmark), it’s back to the programming bit of the interface design, at last. Next step is to figure out how to get ImageJ to write measurements to a file. After that, how to get R to read those measurements out of the same file.

Cuntfickerisch Shiteballs, The Cold, And OmniGraffle

1

Annoying day. Turned more of my asstastic geometric models into OmniGraffle diagrams and .PGM bitmaps. But only got within about half a dozen models of the end. This is taking way too long, and it’s pissing me off. I should be making measurements right now. The clock is ticking like… well, the clock on a timebomb. Another 443 days, and this shit blows up.

Yet somehow not at my prime—trying to get back to active, productive. Went swimming this morning and actually arrived at my desk at a not unreasonable hour, but was tired and distracted all day long. It’s fucking cold outside and I’m pretty sick of it. Aaaaaargh!

Anyway. Tomorrow’s another day, and it’ll be the one I finish these stupid things off and get back to programming.

Back to Interface Design (Hallelujah!)

1

Returned to the lovely fifth floor of the ivory tower, I mean the museum, with a much-improved disposition this morning, having finished the horrifying and vastly unpleasant task of geometric-model-making yesterday. The world seemed a friendlier place as I peered onto the melting snow from my vantage point under the eaves of the museum. It didn’t hurt that it’s 43˚ today (read: it’s above freezing), with a forecast high of 54˚.

The next task—which I imagine will take up a good chunk of the day—is to turn the models I’ve come up with into .PGM bitmap files, which the interface will display when I change from one species to another. This is quite a bit more fun. OmniGraffle!

By lunchtime I had made up three of these figures. I managed to make one more in the afternoon, but succumbed to an intense wave of fatigue and actually fell asleep in the office… When I awoke, I decided that this was probably not the most productive time to continue working, so I decided to call it a successful week (geometric models done!) and return to the interface design with a fresh mind and wakeful body after the long weekend.