District network is still suffering
JANESVILLE The Janesville School District was ill prepared to fight off a computer virus that "ravaged" its computer systems over the past five weeks, a consultant told the Janesville School Board on Tuesday.
Wendy Chretien of Elert & Associates, a consulting company the district hired to critique its operations and efforts to restore the system, was asked to report on the virus attack.
Chretien said:
-- Backup systems were not functioning properly. She recommended a new server backup system to prevent future problems.
-- District computer maintenance staff is "very lean." Staff should have been actively monitoring and maintaining the servers, and that wasn't done.
-- "There is a fair amount of fairly old equipment out there that is just a problem waiting to happen," Chretien said.
-- Anti-virus software did not do its job. If the anti-virus was at 100 percent, that should have blocked the virus, Chretien said.
-- The different computer servers are configured in different ways.
-- The district has no disaster-recovery program.
-- Some systems are not documented, so the only knowledge of them resides "in people's heads."
Chretien did not lay blame for any of these problems. She also did not mention any of the costs of bringing the system up to date.
Chretien said the virus probably came in through an employee's computer, but just who and where has not been discovered.
What is known is that the virus did its damage in about 90 minutes when it attacked Sept. 19.
The result included massive problems for many teachers, students and other staff members, who couldn't access stored information, couldn't update student's grades and couldn't use software applications.
"It just shot through the district," Chretien said.
Most applications were damaged, she said.
Chretien said the virus could have harmed the student information system, financial information and other vital, confidential information, but "the quick actions on the part of one staff member" saved that information.
Brandon Keirns, the district's information technology manager, said after the meeting that all the IT staff members did their parts, but the one who saved that vital information Chretien mentioned was Barrett Jacobsen, a computer technician who was hired this year.
Bunton said he has no doubt the computer system would come out of this episode much stronger.
Chretien said few problems remain to be addressed, and she guessed the system now is about 90 percent repaired. She estimated another month before it's at 100 percent.
Board member Peter Severson gently suggested the district did not communicate well with staff about the problems in the initial weeks, causing frustrations. He said he hoped that the current communications—which include regular districtwide memos—will become a model when dealing with future problems.
Board president DuWayne Severson said he wanted to see a written plan for upgrading the system.

Nov 8, 2008 at 10:57 a.m.
Suggest removal
The earliest viri to infect the home market were on the Apple II. How do I know, I was there and still have an Apple IIc in the garage. :)
http://en.wikipedia.org/wiki/Timeline_of...
Nice summary Techpage. I think, and hope, that one change in education in the 21st century should be the inclusion of Technology Professionals (those who have industry certifications and know how, can do, and really make it their life work) into the Administration role in schools.
The way it is now in Wisconsin, only those who come up through the teacher ranks can achieve administration status in technology.
The only exception I see in this is with a Business Manager. This person needs to have the accounting and business skills first, and then fit into education. They do not have to be a teacher or come up through the teacher path. The same should be for Technology Directors.
Would schools hire a Nurse without being a certified Nurse? I don't know the answer but I assume not.
Would schools hire a Psychologist without being a certified Psychologist? Same question.
I want to make a very clear distinction here. What happens in the classroom should and must be in control of someone who knows how to teach, is a teacher; an Instructional Technology Coordinator. These are the people that know what needs to be taught and have the skills to integrate technology into everyday teaching.
As a tech person, while I care what happens in the classroom, it really isn't up to me to decide, nor do I want to. I want to know what you want to do and then build, design, engineer, support and administer the environment that accomplishes those goals.
Perhaps the easiest way to demonstrate this is in this exact example. Does having antivirus on computer add anything to student learning? What has having antivirus on a computer ever taught a student? Antivirus sits on computers and servers and takes up valuable resources that could be used by other applications. If you answer these questions based on the Instructional Technology Coordinator position, it seems logical to make the argument that spending money on antivirus products is not only a waste but a detriment to student learning. We could take the money we spend on antivirus annually and purchase more smartboard setups.
The true tech perspective adds a dose of real world savvy; this is what you need along with how technology functions, to the story so that things on the back end work and support the overall mission and goal of educating our children. I am confident that if you ask any tech to go without antivirus or a firewall, they will be perplexed.
Nov 5, 2008 at 12:32 a.m.
Suggest removal
Actually I do... and for anyone who's seen office space, it kind of reminds of that.
"I have seven different bosses, Bob"
Nov 1, 2008 at 10:49 a.m.
Suggest removal
On behalf of many school districts IT departments that are under staffed and under paid, I would like to say this to those who do not fully understand what it takes to run an IT department. In most cases you have a few people (if you are lucky) doing the job of many. Some priorities over ride others, it is called to much to do and not enough time to do it! At the least being overwhelmed! It all comes down to budgets and people making decisions on behalf of the IT department and it’s infrastructure, those who do not fully understand what it takes to make it all function. Are they really looking at the whole picture and how one cutback or decision can escalate into another or how quickly the domino effect can happen. These are the individuals that are taking the short cuts and the IT department is left with short end of the stick and having to make it all function no matter what the costs are.
You know that a teacher teaches, a Principal manages/supervises, and the District Administrator does the administrating of the district. Ask yourself do you really know what a Network Administrator does? Do you really understand the whole scope of their role? So who is to blame, really?
Oct 31, 2008 at 1:08 p.m.
Suggest removal
Did we forget about Linux boys? Sorry, couldn't help myself. :)
Oct 31, 2008 at 11:46 a.m.
Suggest removal
Hey Technoguy - I feel the same way about Macs and the previous comment was just silly. However, you are starting to sound a bit jaded. Maybe it's time to move on and involve yourself with an organization that doesn't cut corners.
Oct 31, 2008 at 9:27 a.m.
Suggest removal
Hey Coppertop,(troll)
Where can I pick up that quad core 4 processor MAC sever with raid 5 disk array?
I think you should try the internet and google MAC and Virus and you will fnd your statement is NOT TRUE. It is true that all 8 of you mac users are not a very big target so of course virus writers are ignoring you. However many people need real enterprise level hardware with standardized software to allow information exchange between all the other business out there and unfortunately MS IS THE STANDARD!
Last word on PC vs MAC it's like arguing politics.
Oct 30, 2008 at 10:54 p.m.
Suggest removal
Cripes almighty... get rid of those PeeCee's and buy a Mac! There aren't no stinking virus out there that can affect Macs. Then you can sit by like the Maytag Repair Man waiting for the next call which never comes!
Oct 30, 2008 at 8:03 a.m.
Suggest removal
I am not tryig to make excuses for any of the failures that happened in this situation. I was simply pointing out that most of the time the manpower, time and resources to accomplish the REQUIRED tasks are controlled by people outside the IT environment and that many times decisions that are made with good intentions are the makings of a disaster like the one being experienced by the school system.
I have more times than I care to think about purchased software or learned another operating system so I could use open source software to accomplish NEEED tasks that were not "in the budget".
This should be a valuable lesson learned for everyone involved and I hope the people involved in the decision making process from the school board members all the way down to the voters take away from this incident the lesson that IT is a very complicated and intense environment and it cannot be simply boiled down to a math formula on a spreadsheet.
As for the consulting firm "overseeing" the recovery effort I would bet if I brought any type of consultant into anyones home or business they could create a laundry list of "problems" with any thing we care to inspect.
If the management teams in business would listen to the people they pay every week and really try to understand the needs and reasoning then they might actually be able to make wise decisions in regards to Information Systems.
One of the reasons this is such a problem is data systems are considered a utility and many time when budgets are created a year in advance nobody in the budgeting process can forsee the types of evolutuion in threats that may come up during the year and that makes business types crazy when they crop up and cost truckloads of money to recover from.
WE DEPEND ON DATA SYSTEM FOR EVERYTHING!
This cannot be simplified down to a 3rd grade level discussion at a buget meeting so if you are not willing to understand then excuse yourself from the process
Oct 30, 2008 at 6:28 a.m.
Suggest removal
A district which is underfunded, understaffed, and overworked is no excuse for not having a DR plan. Its common sense!! Operating environment monitoring and health checks should be done weekly. Time to sit down and document manual operating procedures, application disaster recovery procedures, and single points of failure!
Oct 30, 2008 at 6:22 a.m.
Suggest removal
Couple of things.
1. Backup systems do not prevent future problems or anything. They are reactionary for WHEN problems happen. While backup and disaster recovery are good and necessary, this would be my last like of defense. It should be this happened, that happened, and finally, there was no backup or disaster recovery. Look at what happened and fix it!
2. "Very Lean" is a relative term to the value/importance/understanding of what is needed and what could happen. Often, as I see it, IT people are caught up in the tyranny of the urgent and are unable to attend the need of the necessary/essential. Often I am sitting at my computer and people will come up and say "Oh good, you aren't busy..." when I am helping someone else or working on an server/network issue.
I think it comes down to communication and understand among all parties involved, the aptitude, skills, and manpower it takes to do what needs to be done in an information world. It is easy to look at custodial staff and figure out what is needed to clean rooms and make everything work. It is physical and countable. It is much more difficult to do that in the techo/cyber world.
3. Old equipment, that is a problem everywhere. I am asked constantly, "Can we make it work for a little longer", like I have any say in the matter. Truth is, there is MTBF in all electronic equipment. It is not a matter of if it is going to fail, it is a matter of WHEN it is going to fail. There is a reason that HP, Dell, IBM etc will not warranty a server after a certain point. If a system is "mission critial" you better have it under warranty.
4. Virus not 100%. No virus protection is 100%. Something more to this story....
5. Documentation. Again, while necessary, how can you document anything when you are very lean? I am fairly confident the staff was doing all they could do, and then some, with the time they had and there is simply not enough time to do things properly. I am also confident that they took a gamble on what was most important and lost.
6. Virus from an employee laptop....interesting. So this school allowed for personal connections to the internal network. With all of the talk that is out there on security, still it was allowed. No NAC appliance? No other security measures? I hope that other schools take note that while the goal may be for users to bring their own devices into and onto the network, without proper planning and design, disaster can strike.
And I am willing to bet, none of these issues, if ever found out, were intentional.
Oct 30, 2008 at 4:49 a.m.
Suggest removal
Last year the school district had a 7 million dollar surplus, maybe they could have spent a little of it on virus protection.
Oct 29, 2008 at 8:08 p.m.
Suggest removal
tjncj, I'm sure you have some great ideas, but if the money is budgeted for 4 yk, then IT can't get it ... if it's budgeted for hockey - then IT can't get it - That's the (over simplified) way budgets work. And yes, it sucks when it's obvious there is a big need for technology improvements.
We, as customers of this school district have to hold those we elected accountable. And indirectly I hold voters accountable for their choices. If you don't like how this is working - show up at school board meetings and for gosh sakes - vote smart.
Oct 29, 2008 at 6:25 p.m.
Suggest removal
Oh, I voted for the school referendum and I am not calling for anyones job, just some accountability by our administration and school board.
Oct 29, 2008 at 6:23 p.m.
Suggest removal
I have been running or supervising networks for over 20 years starting with Novell 3.12, AS/400, Windows Server, etc. Techno guy I am not questioning your knowledge but I disagree how difficult is it to have a good backup, disaster recovery and virus protection. If your post was the real world businesses and school districts would be crashing for 6 or 8 weeks constantly. It doesn't happen. Protection software catches the vast majority of viruses an other invasions. When they do get through proper backups restore damaged data relatively quickly. In the case of catastophic failure a few days is an eternity, 8 weeks unacceptable. If we have money for Girls Hockey and 4 year old kindergarten we can find dollars to keep our system up, teachers working to there maximum ability and more importantly our children's private data protected.
Oct 29, 2008 at 6:22 p.m.
Suggest removal
its amazing how many people are calling for heads in this situation! Probably the same people who voted over the last couple years (Twice) NOT to allot the school district monies needed to put air conditioners in all the schools, but your willing to spend the kind of money needed described pretty well by technoguy?! I'd bet the farm if it were to come up on a referendum you'd vote "No". As for Viruses.. there is no way a network such as the schools can be 100% protected, even if the virus software is working at peek efficiency. I've worked on many computers that were not on networks, have had virus software working, & they had viruses, spyware, malware, keyloggers, etc. That software dose not stop everything. Kids need to stop treating school computers like they're personal PC, & have some respect for something bought & paid for by someone else.. for the benefit of their future (I know.. not going to happen).
Oct 29, 2008 at 6:17 p.m.
Suggest removal
Just like insurance....it's not worth the price until you need it.
Oct 29, 2008 at 5:22 p.m.
Suggest removal
Technoguy-
Good points about what it takes to test a backup and implement a disaster recovery plan. I think most of us who brought up those points were aware of what it means, but it's still good for others to see it.
That said, it comes down to how important your information is. If you don't care about being down for 2 months while you recover from some unforeseen problem then you can get away without testing your back-ups and implementing DR. If you don't mind permanently losing data, it's not such a big deal and you can get by with minimal preparation. If the data runs your organization, then you better be prepared to pay what it costs to protect it. It seems clear to me the Janesville School District does not value its data. It's probably a common theme in school districts (and many other enterprises).
I think the front line guys here are probably doing an excellent job (and they are probably significantly under-paid too). However, a district the size of Janesville probably shouldn't be running IT "very lean". Of course that's just my opinion. Good luck to everyone involved.
Oct 29, 2008 at 4:24 p.m.
Suggest removal
For all the Monday morning quarterbacks out there here are a few things to keep your minds busy.
Backups
To fully test a server backup you need to have the backup job tape and then you need identical hardware to target with the the backup image. If you do not have a fully functional identical hardware platform to test your backups on you cannot TEST your backups to see if they work. Your only other choice is to try to do the test restore OVER your fully functional server you are using in production and if it does not work now you have no working server and no way to restore it.
Will you put up with having more money put into the budget for Test equipment????
Disaster Recovery Plan
A disaster recovery plan starts with ALL of the upper level managers defining what is important and what order things will need to be restored in. Without these driving decisions there is no way to create a disaster recovery plan! MOST upper level managers out there today do not want to discuss any of this computer stuff because they do not have a clue what any of it is or does. all they know is it costs lots of money and takes lots of manhours to run and they think of computers like light bulbs or heating systems.
When you take a 5% cut from all budgets it looks great that you took thousands of dollars out of the budget but, nobody knows what that means until things like this happen and then all of a sudden "SOMEBODYS HEAD MUST ROLL"
THINK BUDGET CUTS NOW!!!!!!!
Virus protection
Many of the threats out there today ARE NOT VIRUSES there are hundreds of blended threats that cannot be stopped by virus scanners. The simple act of opening an email or clicking on a website link can install dozens of programs on your machine and if that machine is connected to a network then it can spread in minutes to take out every machine on the system. It's not a simple as COMPUTERS 101.
THIS IS NOT YOUR HOME COMPUTER!!!!!!!!!
Budget reductions
Politics
Computer illiterate managers
Staffing cuts
Pay grades for IT staff that are a joke
Managers who "make it" by always finding a way to "make it happen" regardless of the long term effects.
Public outcry that drives all of the above.
NON TECHNICAL people have no business making governing decisions about IT/IS they do not fully understand.
Take your pick from the list above and start there if you are in a blaming mood. Lets not shoot the hardworking guys and gals that have to McGiver together tons of junk and keep it running on bailing wire and chewing gum to allow the systems to work at all.
You should be singing the praise of how many sleepless nights they have spent trying to get this thing back up and running instead of grinding them into dogfood for their efforts!
Oct 29, 2008 at 2:46 p.m.
Suggest removal
I agree with JasonTh. Typical IT staffs are very underfunded. I suspect this is even more common in school districts where every penny not spent directly on instruction is put under a microscope. I know many of us are not surprised by the consultant's findings. That doesn't mean it is ok to go on with the status quo. Even underfunded IT departments have certain things they need to do. A school district even more so; think of how many enterprising young minds could probably have a field day knowing how unsecured the system is.
The real question is what happens next. Clearly IT is not where they can save a few bucks in the future. I also hope other districts in the state learn from this problem. Disaster recovery and good backups should not be considered a "nice to have". Check the backups often. Run DR drills on a regular basis. I hope this stays in the news so we can see what changes are made.
Oct 29, 2008 at 2:03 p.m.
Suggest removal
Everyone is looking for heads to roll, but I think you folks need to wake up and face reality. IT staffs are typically underfunded, understaffed, and overworked. As long as administration has access to emails and files, they typically don't care that the server is seven years old and out of warranty. They don't care that the licenses for anti-virus expired last month and will cost $14,000 to renew...
IT budgets stagnate or get cut because the thousands of dollars in software, hardware, and services that would be purchased for "prevention" and "recovery" are typically the first to be axed.
This leaves the department with just enough money and people to keep the lights on. I don't know the specifics for the JSD information technology department, but the results I've seen are typical of underfunded, understaffed IT shops.
So put down your torches and pitch forks... maybe this kind of disaster will wake up the folks that have been slashing the IT budgets year after year thinking they can save a few bucks.
Oct 29, 2008 at 1:15 p.m.
Suggest removal
I would like to know the additional costs incurred by bringing in an outside consultant! Usually, consultants are very highly paid people!
Oct 29, 2008 at 12:25 p.m.
Suggest removal
I agree with tjncj. Those are all basic practices that any IT manager or Network Admin should have in place and maintain. I think should apply and clean and standardize their network.
Oct 29, 2008 at 12:02 p.m.
Suggest removal
tcncj, I agree that the conditions described are unacceptable (and I've said so before). The board does need to decide how much of that situation was personnel-related and make sure that it isn't repeated.
Oct 29, 2008 at 11:08 a.m.
Suggest removal
It is really disappointing the School board didn't come down hard on Bunton and Keirns. The report by the consultant shows totally unacceptable management of the network. Drop the "gently" and represent the people who voted for you. Hopefully the board is waiting for the right timing administration will react.
Oct 29, 2008 at 10:47 a.m.
Suggest removal
tcnjc - You are correct, this is very bad management. I complained to the school system when they tried to "fix" the problem of a school student getting into the system by suspending them, and I said the network administrator should have been punished, not the student. I never received a response. Apparently, some people believe they don't have to answer to the people that pay their wages. Someone seriously needs to be fired now.
Oct 29, 2008 at 10:44 a.m.
Suggest removal
So, if the system is 90% functional, the other 10% must be everything at Parker?? My kids are reporting that they are unable to do things directly related to their classroom instruction on a daily basis and their education is being compromised each day that this is not corrected.
Oct 29, 2008 at 10:35 a.m.
Suggest removal
No backup system, no documentation, no disaster recovery program, anti-virus system not functioning 100%. Isn't that what they teach the first day in computer networking 101?
Before you post a comment, consider this:
Note: GazetteXtra.com does not condone or review every comment. Read more in our User Policy AgreementPost Comment
Commenting requires registration.