The final day of the public testing component of New York State’s Voting Machine certification testing completed with the tallying and verification of votes cast in the test election. The test plan steps for the day included creating of polling place level reports, verification of the individual machine totals, transfer of machine totals to the Election Management System (EMS), and producing final EMS reports.

For each of the two types of systems being used in New York State, the Dominion ImageCast and the ES&S DS200/Automark, two different ballot layouts were tested, one in portrait and one in landscape orientation. After all of the marked ballot for each test election were scanned, the 4 scanners each produced final tally tapes without incident, matching the expected results. Ballots, some of which were marked by hand and some by BMD, were also manually inspected to verify the results. Then, the cards containing the individual machine data were imported into the EMS system.

Here a problem popped up on the DS200 EMS, which refused to import the election data from the scanner cards. This took some time to figure out, but eventually some careful troubleshooting revealed that the problem was that the order of the candidate names in the EMS database was not set up correctly. Here’s what happened: the EMS system allows the person setting up the ballots to specify one of two ways to order the candidates – by name, or by a specific order. If the operator chooses the “by order” option as was done in the NY testing, they must also go to a separate field and enter a number representing the order: 1 for first, 2 for second, and so on. However, the EMS software makes a classic user interface mistake – it does not REQUIRE them to enter the order number before proceeding. A key user interface principle is that if several related actions must be performed to create a valid result, then the interface must force the user to perform all required steps prior to moving on. So when the test election was first set up last Monday, the “by order” option was specified, but the operator neglected to then put in an order number for each candidate, and the software let them proceed without doing so. Upon import to the EMS, the database refused to accept data which was marked as “by order”, but which had no order numbers assigned. I note that this EMS user interface bug didn’t affect the tabulation on the scanners themselves, but prevented importing of results into the EMS database.

Once the problem’s cause was determined, there was no choice but to go back and recreate the election data in the EMS (this time with the required order number), rescan all the ballots, finally bringing the data back into the EMS. With this fix, the import was successful. This problem will be flagged by the Board of Elections as an issue to DS200 counties, and who will then presumably take care to make sure to enter an order number. But this is an excellent example of how even a small user interface design flaw can make a big difference. The software’s failure to require that the user enter the order number when setting up the election BEFORE voting, leads to a problem when an attempt is made to bring results back in to the systems AFTER the election. Had this happened in a real election, the county would not have been able to use the EMS system to tabulate and report results. Instead, they would have had to manually tally machine tapes and hand count paper ballots – not such a bad thing if you ask me.

At the end of the day, all the results from the four systems matched the expected results, and the weeklong public component of New York State’s certification testing came to a close. Note that last week’s public testing is only one part of a much larger set of certification testing being conducted by New York State, and as such had relatively limited goals – primarily a public demonstration of a complete end to end test run of both systems.

Later this week, I’ll post my analysis of the week’s testing, how it fits into New York State’s larger testing procedures, and thoughts on what it all means for New York State voters.