The November 2022 election in Prince William County used am electronic pollbook made by DemTech, which recently had a merger with Civix. Pollbooks are the devices the election officers use to look up a voter at the voting location, and check them in so they can receive a ballot. Ninety-four counties and cities in Virginia used the DemTech pollbook system.
In Prince William, at least 70 out of 103 precincts had issues with these pollbooks. Chief officers were given cursory instruction by the county and the at least 25 precincts had wrong passwords to login. The county has no formal help desk, and calls made to headquarters by the chiefs were not logged. The numerous problems were highlighted in a Request for Technical Review submitted by the Prince William County Election Integrity Working Group on January 27, 2023. In summary, and this is not all of the problems:
o USBs had to be removed and reinserted for pollbooks to begin functioning.
Chief Notes from 2022 Election, and Request for Prince William County Elections Office Technical Review and
o Three precincts had missing parts in their pollbook kits.
o File exports could not be completed.
o Address locators not working with multiple EPBs and at eight precincts.
o Two precincts had pollbooks auto updating during election day and subsequently going offline or shutting down.
o Had to turn off EPBs at polls close to logout.
o Nine precincts had pollbooks stopped functioning entire. They, ‘froze’ suddenly and had to be started and logged into again.
o Pollbook scanner devices pulling up wrong people or addresses, and within the
same precinct another EPB finds the correct person with the scanner.
o Ten precincts had pollbooks not connecting to the rest of the system, ie, the Merlins.
o Dates and times incorrect across multiple pollbooks in at least five precincts.
o More pollbooks showing logged in at a precinct than physically present
o Pollbook lag: At least six precincts noted voter entries disappearing before a proper look up of a voter could be made, or slowness of pollbooks response.
o Unknown administrator logged into EPBs, preventing logout at the end of the day.
o Office of Elections had to shut down EPBs at polls close at several precincts and numerous precincts had to do a hard shutoff because the system would not logout per normal procedure.
o Pollbook count was off count off all day without user error at at least twenty-four precincts.
o Start of day EPB count not zero at one precinct.
Root Cause Analysis of DemTech ePollTAB Precinct Management System, submitted January 27, 2023 by the Prince William County Election Integrity Working Group.
Our request for technical review has gone unanswered and not acknowledged by the county. However it also was handed to the Prince William County General Registrar at the February Electoral Board meeting.
At the March 7, 2023 State Board of Elections Meeting, a report was handed into the state board with some most interesting paragraphs pertaining to a couple of pollbook problems. Page 58 of this report says,
DemTech Electronic Pollbooks
Pg 58, https://townhall.virginia.gov/l/GetFile.cfm?File=meeting\151\37224\Agenda_ELECT_37224_v3.pdf
As of 2019, Virginia is one of thirteen states that certify electronic pollbooks (EPBs). In the
Commonwealth, localities have over 8,000 pollbooks supported by four vendors: Election Systems and Software, Knowink, Dem Tech, and Robis Elections. DemTech is the most widely used electronic pollbook in Virginia, with 94 localities as customers.
While no issues occurred with DemTech electronic pollbooks during the 45 days of early voting,
some localities had issues with their DemTech pollbooks on Election Day. Several factors
contributed to these issues. The primary cause was a malfunctioning time chip in some units that caused the EPBs to slow down, which required them to be restarted. The secondary issue was a de minimis change approved update that allowed the key, which refers to the USB flash drive used in unlocking the EPB for use, to remain in the unit. Users had to pull out and reinsert the key to open the EPB, which resulted in confusion for some officers of election.
Localities were delayed in purchasing the upgraded EPBs, shrinking the amount of time to train
and understand the upgrades. One general registrar noted the impact saying, “[L]ocalities were
scrambling to get the new software and receive training, so they could pass the information on the to their officers of elections.” Despite these issues, all data was found to be secure and accurate on the EPBs after an analysis of the audit logs and VERIS logs was conducted.
DemTech will provide new time chips to all customers to ensure the time and date are correct
when data is entered into VERIS along with a detailed instructions on how to install and
synchronize the chip. Training for both DemTech employees and localities will be provided for
better understanding of the system and how to communicate issues and solutions. DemTech is
making available online training for all localities using their EPBs.
KEY TAKEAWAY: DemTech, ELECT, and localities are working together to ensure that these
issues do not occur in future elections.
The above may seem like a thorough response to someone new to election technology. Sorry to say that what is submitted was wholly inadequate. The first questions are;
- WHAT TIME CHIP?
- Time chips are in software or on a motherboard.
- What device houses this time chip?
- How is DemTech providing new time chips for localities to install and synchronize?
- What election office has technical staff to install chips on a motherboard, over 300 of them in the case of Prince William?
Why is this important? First, the county paid over $300,000 for these new pollbooks. Second, the pollbook house our voter rolls during the election and this is where ballot distribution to each voter is entered for the county and state records which are required by state and federal law. Third, the state process for pollbook certification is now proven to be subpar and disastrous.
Tom Hurt, who is a world class expert in system assurance explained,
“The time chip does not slow down on its own. It operates at a hardware-based frequency and software (SW) interprets the frequency to produce wall-clock time. The SW might reach out over the Internet to any of a hundred sources to set the clock for the current time. Maybe this chip was the time-source for the device. Or, The BIOS maintains the time and this chip is set to that time by SW so the chip sends back current time in real-time to any SW module requesting it. If the chip had the onboard SW (BIOS), and time slowed down, then the trouble is huge…There is no reason to ever change any SW in a time chip or in BIOS to modify time processing. In a PC, when something stops working, the problem is and caused by software.”
Here is a truncated version of how a pollbook gets approved for election use in the Commonwealth. When a vendor creates a pollbook system to be used in an election, it applies to the Commonwealth. The Commonwealth provides a certification standard that the equipment has to meet through successfully passing a series of test cases to show it meets the functionality and requirements of the law during the election. Once all test cases are completely and the system passes, a report is sent to the Commonwealth’s Voting System Program Manager. If she is satisfied she submits a recommendation for approval to the State Board of Elections, which commonly votes positively based on the Voting System Manager’s recommendation. Once approved only that system version can be used unless the vendor applies for a system change approval.
Here is the ProV&V test report submitted as part of the application for de minimis change.
Here is the approved de minimis change from version 1.0 to 1.2
In the case of DemTech, the original system approval was version 1.0 on September 3, 2020. Later in 2022 DemTech submitted a ‘de minimis change,’ meaning one that did not require a full recertification but simply a change approval because the changes were minor. This de minimis change was approved by the Commonwealth’s Commissioner of Elections on April 21, 2022. It brought the system to version 1.2, and went through despite the mention of internal changes on the application. This is the de minimis change referenced in the State Board Report and provided above.
Again according to Tom Hurt,
The second problem they admit to “secondary issue was a de minimus change approved update that allowed the key, which refers to the USB flash drive used in unlocking the EPB for use, to remain in the unit. Users had to pull out and reinsert the key to open the EPB, which resulted in confusion for some officers of election” If a flash drive unlocks the EPB, the action happens via software. Period.
No change to software can be de minimis. Any change to software must be regression tested to 1) prove that the problem is fixed, 2) prove that only the problem was fixed, and 3) proved they didn’t break something else.
Pulling out a USB stick causes software to operate in the operating system to assure certain operations and data. When the USB stick is inserted, onboard software in the USB chip tells the operating system (OS) what it is. The software in the USB stick was not changed, so someone changed the software in the OS or some OS module in the device to allow the key to remain inserted.
The flow of control proceeds in the CPU instruction-by-instruction through all instructions in the set of software modules. If there is software that isn’t executed, it is dead code and a severe vulnerability (adversaries hide their subversions as dead code that only becomes active when a trigger signal is received from the Internet). Recognizing the above key is a software transaction and someone changed the software.
The change from version 1.0 to 1.2 was NOT de minimis, and required a new certification. There is no system assurance that the system will perform as intended or apparent plans for such assurance testing.
Still no response from Prince William County on what they have found to be their deficiencies in the pollbook deployment, or on any aspect of our request for technical review.
UPDATE: The ‘time chips’ are actually real time clocks shipped by the DemTech and installed by the election office this past Spring 2023. The new real time clocks used for the 2023 primary still did not resolve the problems experienced in 2022.