S.F. officials locked out of computer network
Jaxon Van Derbeken, Chronicle Staff Writer
Tuesday, July 15, 2008
(07-14) 19:23 PDT SAN FRANCISCO -- A disgruntled city computer engineer has virtually commandeered San Francisco's new multimillion-dollar computer network, altering it to deny access to top administrators even as he sits in jail on $5 million bail, authorities said Monday.
Terry Childs, a 43-year-old computer network administrator who lives in Pittsburg, has been charged with four counts of computer tampering and is scheduled to be arraigned today.
Prosecutors say Childs, who works in the Department of Technology at a base salary of just over $126,000, tampered with the city's new FiberWAN (Wide Area Network), where records such as officials' e-mails, city payroll files, confidential law enforcement documents and jail inmates' bookings are stored.
Childs created a password that granted him exclusive access to the system, authorities said. He initially gave pass codes to police, but they didn't work. When pressed, Childs refused to divulge the real code even when threatened with arrest, they said.
He was taken into custody Sunday. City officials said late Monday that they had made some headway into cracking his pass codes and regaining access to the system.
Childs has worked for the city for about five years. One official with knowledge of the case said he had been disciplined on the job in recent months for poor performance and that his supervisors had tried to fire him.
"They weren't able to do it - this was kind of his insurance policy," said the official, speaking on condition of anonymity because the attempted firing was a personnel matter.
Authorities say Childs began tampering with the computer system June 20. The damage is still being assessed, but authorities say undoing his denial of access to other system administrators could cost millions of dollars.
Officials also said they feared that although Childs is in jail, he may have enabled a third party to access the system by telephone or other electronic device and order the destruction of hundreds of thousands of sensitive documents.
Authorities have searched Childs' home and car for a device that could be used in such an attack, but so far no such evidence has been found.
As part of his alleged sabotage, Childs engineered a tracing system to monitor what other administrators were saying and doing related to his personnel case, law enforcement officials said.
Childs became the target of suspicions inside the technology agency this year, and the case was referred for police investigation in late June, authorities say.
At a news conference announcing Childs' arrest, District Attorney Kamala Harris was tightlipped about what his motive may have been.
"Motive is not necessarily an element of a crime," Harris said. "This city employee committed four felonies."
She added, "This involves compromising a public system that we rely on. Its integrity has been compromised."
The system continues to operate even though administrators have limited or no access, officials said.
"Right now our system is up and running and we haven't had any problems so far," said Ron Vinson, chief administrative officer for the Department of Technology.
Vinson said the city is "working around the clock" to make sure the system is maintained and operable.
Nathan Ballard, a spokesman for Mayor Gavin Newsom, said the mayor was "confident that (the Department of Technology) is doing everything necessary to maintain the integrity of the city's computer networks."
Childs appeared in court Monday but did not have a lawyer assigned to him.
Childs, according to payroll records, earned $126,735 in base pay in 2007 and additional premium pay of $22,534, for a total of $149,269. Vinson said the extra money was apparently compensation for being on-call as a trouble-shooter.
E-mail Jaxon Van Derbeken at firstname.lastname@example.org.
lame title change
I got it from AP, i swear!
Four felonies?? WTF?
For fucking up their puter? Give me a break.
There has got to be more to the story!
I guess he took the red pill.
That's nothing. I used to work for the City and County of San Francisco Department of Building Inspection. I was the Principal Network Engineer. That city is corrupt more than you'll ever know. Here's my old boss:
SAN FRANCISCO (Feb. 10, 2003)—City Attorney Dennis Herrera filed a major public corruption lawsuit today charging Marcus O. Armstrong, the San Francisco Department of Building Inspection’s top technology official, with an elaborate kickback and illegal payment scheme that defrauded City taxpayers of more than one-half million dollars over the last two years. Also named as defendants in the suit are now-defunct technology vendor Government Computer Sales, Inc.; the company’s former chief executive officer and owner, Robert Fowler; and Foster City, Calif.-based technology consultant Raman Kumar.
"In masterminding an elaborate scheme intended for their own self-enrichment, Mr. Armstrong and his cronies have betrayed a public trust and cost our City something more than the money it sorely lacks," Herrera said. "Public corruption diminishes the confidence of our citizens in their government and dishonors the hard work provided by honest public servants every day. It’s a slap in the face not just to taxpayers, but to firefighters and cops, nurses and teachers. They all deserve better—and wrongs such as these deserve the most aggressive civil remedies and damages our office can obtain."
Herrera’s civil lawsuit results from a yearlong investigation by the City Attorney’s Public Integrity Task Force—a specialized unit he created after taking office in January 2001—working in tandem with auditors from the office of City Controller Edward Harrington. Seeking substantial penalties and punitive damages in addition to the recovery of stolen funds, the suit alleges a complex web of wrongdoing that includes fraud, conversion, unfair business practices, false claims, breach of contract and violations of both state and local conflict-of-interest laws. A parallel criminal investigation—which is likely to involve interstate activities—has been referred to U.S. Attorney Kevin Ryan and the FBI, with whom Herrera has pledged his office’s full cooperation.
According to the suit filed in San Francisco Superior Court today, the illicit kickback scheme began more than two years ago when Armstrong, information technology director for the Department of Building Inspection, pushed for Government Computer Sales, Inc. to be selected as the technology vendor on three major projects to improve the department’s services to members of the general public. Hidden from City officials at the time, however, was the company’s virtual insolvency: in the process of defaulting to creditors for more than $16 million, Government Computer Sales’ CEO, Robert Fowler, began illegally transferring company assets offshore—to a Caribbean bank Fowler himself controlled—even as the company was assuring City officials of its intention to fulfill its contract with the department. It never did.
Instead, Armstrong intentionally misled superiors and lied to city auditors to gain approval for what were, in fact, illegal advance payments to Government Computer Sales that would eventually total more than $500,000. The company’s work on the projects, which was incomplete at the time, would never be completed. And though most of Government Computer Sales’ subcontractors would go unpaid for the projects, one subcontractor who did receive significant payments from the hemorrhaging company was Raman Kumar.
It was no lucky coincidence: according to the City Attorney’s complaint, Kumar would kick back more than $21,000 of his payments from Government Computer Sales—usually within days of receiving them—in business checks written to "Mindstorm Technologies" and "Monarch Enterprises," both phantom front companies set-up and controlled by Marcus Armstrong. Among Armstrong’s ill-gotten gains from the kickback scheme, according to the complaint, is a 2002 Mercedes-Benz automobile.
"When I set up the Public Integrity Task Force less than a year ago, I envisioned it as an innovative, multidisciplinary vehicle for civil law enforcement enabling us to aggressively pursue those who would violate the public trust," Herrera said. "Today’s lawsuit furthers that vision, and I’m enormously grateful to Chief of Public Integrity Lori Giorgi and Chief of Investigations Timothy Armistead for their dedication and professionalism—not merely on this case, but on all the cases that will continue to strengthen public integrity in our City."
That explains it, alright.
UPDATE: guy gives up passwords
San Francisco's mayor gets back keys to the network
IT administrator Terry Childs is in jail for previously refusing to hand over the admin passwords to the city's multimillion dollar WAN
By Robert McMillan and Paul Venezia, IDG News Service
July 23, 2008
San Francisco Mayor Gavin Newsom met with jailed IT administrator Terry Childs Monday, convincing him to hand over the administrative passwords to the city's multimillion dollar wide area network.
Childs made headlines last week when he was arrested and charged with four counts of computer tampering, after he refused to give over passwords to the Cisco Systems switches and routers used on the city's FiberWAN network, which carries about 60 percent of the municipal government's network traffic. Childs, who managed the network before his arrest, has been locked up in the county jail since July 13.
On Monday afternoon, he handed the passwords over to Mayor Newsom, who was "the only person he felt he could trust," according to a declaration filed in court by his attorney, Erin Crane. Newsom is ultimately responsible for the Department of Telecommunications and Information Services (DTIS) where Childs worked for the past five years
Mayor Newsom secured the passwords without first telling DTIS about his meeting with Childs, according to DTIS chief administrative officer Ron Vinson, who added, "We're very happy the mayor embarked on his clandestine mission."
The department now has full administrative control of the network, he said in an interview Tuesday night.
It's likely that Childs had a lot to tell the mayor when the two met.
Childs' attorney has asked the judge to reduce Childs $5 million bail bond, describing her client as a man who felt himself surrounded by incompetents and supervised by a manager who he felt was undermining his work.
"None of the persons who requested the password information from Mr. Childs ... were qualified to have it," she said in a court filing.
Childs intends to disprove the charges against him but also "expose the utter mismanagement, negligence, and corruption at DTIS, which if left unchecked, will in fact place the City of San Francisco in danger," his motion reads.
Vinson dismissed the allegations. "In Terry Childs' mind, obviously he thinks the network is his, but it's not. It's the taxpayers'," he said. "The reason he's been sitting in jail is because he denied the department and others access to the system."
The court filings help explain just how this happened.
According to an affidavit from James Ramsey, an inspector with the San Francisco Police Department, he and other investigators discovered dial-up and DSL (digital subscriber line) modems that would allow an unauthorized connection to the FiberWAN. He also found that Childs had configured several of the Cisco devices with a command that would erase critical configuration data in the event that anyone tried to restore administrative access to the devices, something Ramsey saw as dangerous because no backup configuration files could be found.
This command, called a No Service Password Recovery is often used by engineers to add an extra level of security to networks, said Mike Chase, regional director of engineering with FusionStorm, an IT services provider that supports Cisco products.
But without access to either Childs' passwords or the backup configuration files, administrators would have to essentially re-configure their entire network, an error-prone and time-consuming possibility, Chase said. "It's basically like playing 3D chess," he said. "In that situation, you're stuck interviewing everybody at every site getting anecdotal stories of who's connected to what. And then you're guaranteed to miss something."
Without the passwords, the network would still continue to run, but it would be impossible to reconfigure the equipment. The only way to restore these devices to a manageable state would be to knock them offline and then reconfigure them, something that would take weeks or months to complete, disrupt service, and cost the city "hundreds of thousands, if not millions of dollars," Ramsey claims.
Crane argues that these monitoring devices were installed with management's permission and were critical to the smooth functioning of the network. They would page Childs when the system went down and allow him to remotely access the network from his personal computer in case of an emergency.
In interviews, current and former DTIS staffers describe Childs as a well respected co-worker who may have gone too far under the pressure of working in a department that had been demoralized and drastically cut as the city moved forward with plans to decentralize IT operations.
About 200 of the department's 350 IT positions had been cut since 2000, mostly to be relocated to other divisions within city government, said Richard Isen, IT chapter president with Childs' union, the International Federation of Professional and Technical Engineers, Local 21.
Despite his conflict with some in the department, Childs has a lot of support there, Isen said. "There is a lot of sympathy, only because there is a basic feeling that management misunderstand what we actually do and doesn't appreciate the complexity of the work."
(Paul Venezia is Senior Contributing Editor with InfoWorld)
Well, shit. I felt like that at NCTAMS, but that didn't give me justification to sabotage PACFLTs communications.
The No Service Password Recovery explains a lot.
Fantastic backup strategy, BTW.