The fourth day of New York State’s Voting Machine Certification testing started with marking of the test ballots. Because of the nature of this testing, the votes on the ballots are predetermined and marked accordingly. This is necessary for this type of test because in order to verify the output (in this case the final vote totals), you have to know what the inputs were (the marked ballots). Members of the Citizen Advisory Committee, of which I am a member, and State Board staffers marked several hundred ballots. The voting patterns are designed to test various possible permutations – undervotes, write-ins, vote for three out of five, and so on. While we were marking ballots by hand, the teams were creating ballots marked by each of the BMDs as well, so that the test deck contained both hand and BMD ballots.

After the ballots were marked, the teams began the slow process of inserting ballots. This turned out to be quite time consuming, given that each ballot insertion proceeded through multiple test steps, with each step being checked off on the spreadsheet when completed.

The Automark BMD being used to test landscape ballots immediately showed a problem, apparently not recognizing the ballot as valid when inserted. The testing teams went into troubleshooting mode, and by methodically trying out different types of ballots, and a different machine, it was finally determined that the layout of the landscape ballots was at fault. Apparently, the landscape ballot design had the header (which displays information about the Election District in human readable form) printed right next to the timing marks (the little black squares printed on the outside edges of the ballots which the devices use to determine the ballot style and marking locations). The working theory is that having the dark type right next to the timing marks made it impossible for the Automark to recognize the ballot. The portrait oriented ballots, where the header is not in close proximity to the timing marks, did not display this problem, one of the clues the team used to determine what was going on. This demonstrates one of the values of this type of detailed testing. We’ve now learned that landscape oriented ballots for these systems must have the header information placed somewhere else (there’s plenty of room in other areas of the ballot). Imagine if this had not been discovered – landscape ballots in the November election, if laid out in this fashion and used for the Automark BMD, would not have worked.

My only gripe with the test plan so far is that I’d like a lot more test ballots than currently called for. Each of the four systems will have approximately 64 ballots tested. This feels like way too small a number to me, and I would be happier with a much larger test deck. The State Board sought a balance between a sufficient number of ballots to test versus the time, and expense, of testing more (recall that each day of testing ties up the Board’s entire operations staff and paid consultants from SysTest and NYSTEC). I can understand the need to find a balance, but I still would be more comfortable with a larger number of ballots. On the other hand, I’m hard pressed to come up with the correct number. Would 100 test ballots tell us anything more than 65? How about 200, or 300? Ultimately, what I really would like to have seen is a full mock election, where ballots are voted throughout the entire day on multiple machines. But this was not mandated by the State Board as part of the certification regimen, and I knew in advance the goals of a functional test like this are to demonstrate complete and correct end to end operation of the systems, and shake out any problems that pop up. I’m going to do a bit of research over the weekend to see what a ‘correct’ number of ballots for a thorough functional test might be. If you’ve got an opinion, send me an email with your thinking on this.

Now I’m off to the final day of testing, when the results from the scanners are brought back into the EMS systems, and the tallies checked against the expected totals. Also, final procedures for closing out elections and producing various post election reports will be part of the day’s events. I’ll post again about the final day of testing on Saturday morning, and do an analysis of the entire process next week. Stay tuned.