Fragile Tests (ultra-wonkish)

An importance part of Tinderbox development is a fairly big suite of tests that run all the time. Right now, there are 1,297 tests, and they’re run many times a day. Tests can’t check everything, but they do check lots of things, and let me know when something comes unstuck. I seldom let a test fail for more than a few minutes.

A fragile test is a test that sometimes passes but occasionally fails. That’s a very bad thing. Most of the fragile tests are fragile for well-understood reasons; for example, there are a couple of tests for time intervals that fail just before or just after we switch to daylight savings time. Those could be fixed, of course, but it’s just simpler to leave myself a note.

Tests that involve multiple cores at the same time can be fragile because you never know exactly how many cores will be available, or when one of them might be interrupted. Any sign of fragility is an asynchronous test is almost surely a serious failure.

Sometimes, though, a test is fragile without rhyme or reason. Here’s one saga.

Tinderbox’s has a bunch of kibitzers that help keep the map view neat. For example, the Top Alignment Kibbitzer keep an eye out for notes whose tops are nearly aligned; if you move a note so its top is almost but not quite aligned with its neighbor, the Top Alignment Kibbitzer will align it. There are a whole bunch of kibbitzers, and they're not hard to test: you put down a note, you move a second notes so the kibbitzer in question should wake up, and you see if the kibbitzer does in fact wake up and check that it does the right thing. You also test that other kibbitzers don't wake up at inappropriate moments and start shoving things around for no good reason.

We’ve got about 40 of these tests, and one of those tests has been fragile for months. It tests the Top Adornment Spacing Kibbitzer, an advisor that tries to move notes that are near the top of an adornment a consistent distance from the top edge. This would fail about once a week, without rhyme or reason. Only the top adornment spacing test failed; the test for the Left Adornment Spacing Kibbitzer never gave any trouble. What was wrong?

The problem, it turns out, was a bad C++ copy constructor for LayoutInfo objects — the objects that keep track of layout details in the map view. LayoutInfo has a ton of instance variables — it’s really just a bundle of data and nothing else. One of those details is the size of the text for drawing the title.

Now, the title of an adornment is drawn at the top of the adornment, and it's handy to have the Top Adornment Spacing Kibbitzer try to reserve the top area of the adornment for the title. That’s reasonable! But, somehow or other, the text size got dropped from the copy constructor, and the Kibbitzer happens to use the copy constructor when it’s trying to decide whether to wake the kibbitzer. The test expects the text size to be zero, and usually it actually was zero, but once in a while you'd get a garbage value and the kibbitzer would fail to fire because the adornment was drawing the title in billion point Helvetica. The baseline is somewhere beneath the basement, and the kibbitzer goes back to sleep because there's no way this can be anything it can fix.


  • Fragile tests are fragile for a reason.
  • C++ copy constructors are tricky critters.

I was in my freshman-year dorm room. It was late, probably approaching midnight. I was working through nucleophilic aromatic substitution in Morrison and Boyd when one of the sophomore women in the triple across the hall shouted, cheerily but quite loudly:

I don’t have time to have sex: I’ve got Biochemistry!

“Yep,” I nodded to myself as I cheerfully turned back to organic chemistry. “That pretty much sums up this place.” David Mamet once referred to his alma mater as “sex camp”; Swarthmore was Liberal Arts camp where sex replaced archery and canoeing.

The best part of Lev Grossman’s book trilogy The Magicians is a subplot in which magic — the wonders of Harry Potter and Narnia — is seen from the outside as a sort of miraculous heroin rush, an infusion of power and force that makes everything wondrous. Grossman’s world has a college for magic — Brakebills — but magic is studied with even greater intensity and desperation in a network of dingy, underground safe houses populated by addicts, criminals, perverts, demons, and other rejects who won’t, or can’t, give up the dream.

The television adaptation (SYFY/Netflix) takes this metaphor and extends it vastly, turning it into an extended study of five young New Yorkers and their reaction to higher education and all that entails. We have, for example, Kady, a tough girl from the Bronx who is as smart and accomplished as anybody, and if she’s had to do some things she’s not proud of, if she’s had to commit some crimes, even, it was only what she had to do to save her mother and to survive. The school, on the other hand, sees a character defect, and out she goes.

Or, take Penny. In the books, he’s Ron Weasley with the serial numbers filed off, and he exists to show us that the Potter-Weasley friendship is an illusion: in realist mode, those boys would have always hated each other. In the adaptation, Penny is South Asian, aloof and always watching from the outside; he’s cool because he’s always his own spectator. He sees the faculty clearly, and sees that to them he’s just another part of another class, and next year there’s be yet another class so, for them, there’s little point in opening their eyes. He knows the teachers are wrong.

Margo is Manhattan: rude, greedy, arrogant, destructive, and indomitably effective. Elliot’s the tormented, talented aesthete who moved to New York from some comfortable hellhole in the Midwest. Alice’s parents are academics who probably met at Columbia (or possibly CUNY), now teach in the Berkshires, read all the best child-rearing books and never had time to figure Alice out. The hero, Quentin Coldwater, is just a shlump who does well in school; he means well but seldom has any clear idea of what to do or how to act.

Then there’s Julia, who was Quentin’s girl next door, who was always just a little better than Quentin at school and at life. But when they apply to grad school, she inexplicably fails the Brakebills entrance exam. Once Julia knows that magic is real and Brakebills exists, her fallback school (Yale) is worthless, and Julia dives into the world of safe houses to learn what Higher Education denied her. She dreams, endlessly, that someday the Dean will recognize that they made a mistake and will welcome her to grad school after all.

All these people are deeply, deeply enmeshed in the study of magic. It consumes them — in one case, literally. It defines their life. It’s totally grad school. Grad school problems are all over the place: Elliot, for example, lands a really good job (High King) before graduation, and trying to finish his thesis while holding down the job of his dreams is tearing him apart.

When it’s just you and the molecule and it’s not going well, the world can be a very bleak place.

Uniquely, the series captures the way this can be so differently true while winding up with the same driving obsession. Quentin loves magic; fair enough. Margo loves its power. Elliot adores beauty and detests unnecessary work; for him, magic is the best way to a really good Amatriciana and an even better wine to pair it with. And Julia: Julia just has to know stuff: how it works, how to do it, and what’s behind the curtain.

Race, gender, class: it’s all there, but we don’t need it for differentiation and, even if we weren’t divided by them, we still wouldn’t be on the same page. Yet to the greater world, they’re all this tiny cadre: esoteric, incomprehensible, privileged and interchangeable.

Jan 18 17 2018

iMac Pro

The new iMac Pro builds Tinderbox from scratch in 30sec — about 6x my trusty MacBook Pro 15. In other respects, it’s just fast, but for compiling and running code, it’s blazing. The biggest performance leap I’ve seen in a long time, and worth the inconvenience of losing portability.

The keyboard is driving me nuts, but we’ll sort that out eventually.

by Lev Grossman

An early book by the author of The Magicians, this thriller thrusts a timid and rather bored young Wall Street broker into the race to locate a mysterious ancient book. There’s an immersive video game in the mix, too — one that seems peculiarly tailored to the pursuit of this tome, and there’s an icy but beautiful young scholar, a beguiling duchess, and a rent-controlled Manhattan apartment. A castle, too. What’s not here, unfortunately, is the wonderful mix of street grit and wonder that occasionally illuminates The Magicians; too often, this is just an American Possession without the scholarship.