Cyber attack update provided by City Manager

Cyber attack update provided by City Manager, John Collin at the Common Council meeting on Monday, January 11, 2021.

Members of Council.

This evening I will provide you a rather lengthy update on our ongoing restoration efforts post cyber attack. To ensure that I cover everything necessary, I will be speaking from a prepared script and will be pleased to take questions thereafter.

By way of background, you will recall that on the 13th of November 2020, hostile actors (criminals) executed a ransomware attack on the city's IT systems. Upon discovery, we immediately severed our IT links to the outside world in order to ensure that we, the City, did not cause the virus to spread. I am happy to report that we have no indications whatsoever, that there was any spread of the ransomware from any City-owned assets or systems to others.

Since the beginning of this incident, we have kept the public informed of what they need to know, as we know it. Today, we continue that approach with this briefing. Although we are fully committed to keeping the public informed and providing what everyone needs to know, we are equally committed to protecting the interests of the City. We have been consistent in our information passage. We do not divulge information that could be useful to those who attacked us. This includes giving them nothing on what systems they successfully compromised, how we contained the virus or how we are mitigating potential future attacks. It also includes not discussing specific ransom demands in public or our decision criteria for such demands. In short, we do not want to give these criminals any information that could help refine their tactics and techniques nor do we wish to provide such information to copycats or future wannabe hackers.

Moreover, there is an active police investigation on this attack which provides further reason to not divulge certain information.

That said, we have, and will continue to keep the public informed if there is any compromise of personal information and on any impacts to our delivery of services. We will also, in due course, outline what this incident has cost the City financially.

Looking at the question of the possible compromise of personal identifying information (often referred to as PII and includes such things as credit card numbers, bank account details and Social Insurance Numbers), we did hire, early on, a third-party expert-company to conduct a forensic analysis to attempt to determine if any such information has been compromised. Although we have yet to receive the final report, indications so far are that no PII has been leaked or stolen. We do not expect this to change in the final parts of the forensic analysis but even if it did, I would point out that the City retains very little personal information on residents or businesses since most of our needs are satisfied through Cloud-based applications therefore we do not store this information within our networks. Our own employees are at slightly higher risk since we do retain some of their information but again I stress, that our forensic analysis has not identified any compromised PII at this time.

For everyone, we should all not only be concerned about the cyber attack on the City but also the many other attacks that occur literally on a daily basis. It is therefore good practice to regularly check bank accounts and other financial statements for discrepancies or suspicious activity and report anything unusual to your financial institutions as applicable.

After we successfully severed our systems’ ties to the outside world on 14 November, we turned our attention to eradicating the virus and restoring some form of interim capability. The interim capability was necessary since we disconnected and shut down our normal systems out of an abundance of caution. I am pleased to confirm that we relatively quickly re-established a temporary website to provide information to our community, recreated email and phone accounts in order to communicate, and developed workarounds for other critical IT functions that we had shut down. As one example of this, we develop alternative IT processes to ensure all employees continued to be paid. We did the same for most customer services and I will speak more about his in a few moments.

We continue to improve services on our interim network and we work towards full restoration. Our analysis has confirmed that the degree of penetration of the virus was indeed extensive. We have concluded for this, and several other reasons, that REPAIR of our existing systems is not the best choice. Rather, we have decided to build an entirely new network. Not only will this afford us the opportunity to take advantage of all of the latest innovations in cyber security and network design, it will also remove the risk of virus remnants that can occur in a repaired system.

To build a new network will take time. Our workplan sees incremental restoration with the bulk of the project being complete in the April/May timeframe. We will take our time to get this right. We must not rush. But I must tell you that to rebuild everything over a 4-6 month period is still very ambitious.

Until various functions of our new system come on-line, we will continue to rely on and improve our interim capabilities. Our temporary website continues to add additional important data and within a few weeks, our new permanent website will be launched. We have developed processes whereby all public meetings can continue (such as the PAC held last week) and we restored a few days ago the ability to take credit card and debit payments for bills owed.

Generally speaking, almost all municipal services continued despite the cyber attack. This includes but is not limited to: all emergency response (police, fire, paramedic and EMO); garbage removal, provision of water and treatment of sewage, road repair, winter storm management, public transit, Council meetings, maintenance activity, customer service help desk, and general public works. The credit for this rests with the entire team that worked tirelessly to make this so. Where there have been temporary suspensions, in most cases, alternative approaches are now in place and most disruptions were very short in duration – such as the few days where building permits were delayed.

There are still a few services to be restored. For example, although safe clean drinking water was never at risk, there have been no metered water bills issued since the middle of November. But, we have now found a temporary solution and water bills should be issued this week. We only have a limited ability to issue parking tickets but this too will soon improve. We have had challenges providing support for land transactions, but we expect to have this issue resolved shortly.

In short, there have only been minor disruptions to customer services and we remain optimistic that this will remain so and we build our new network. The most significant impact of the attack has actually been on the staff and employees since our internal processes have been changed and we will continue to face challenges until our new network is built and fully functioning in 4-6 months’ time.

Turning to costs, I am pleased to inform Council that we have a comprehensive cyber insurance policy in place. There are, of course, deductible amounts but these are manageable. We are in close coordination with our insurance company to ensure that their requirements are satisfied as we move through restoration process, and are confident that the majority of our response efforts will fit within the policy terms. Generally speaking, our insurance will only cover the return of our previous capabilities. Since we are building new and adding additional enhancements, the costs of the improvements will be borne by the City. This too is manageable since we have a sizeable IT reserve fund and we were planning to adopt these enhancements in the coming few years in any event. I must emphasize that this incident, much like COVID 19, has shown us again the value of having reserve funds. We will NOT need to adjust any yearly budget or alter our service delivery to our community because of this attack. We can use reserve funding to cover the expenses instead of cutting costs through elimination of services – exactly what reserves are designed to do.

Although we can approximate some of the public costs, we are not yet ready to describe the entire costs in detail since we are still working through the restoration plan. We will return to Council with the exact costs to our public, once they are confirmed.

I also wanted to address a rumour on social media that the City has paid a ransom. We have not. Any decision to pay a ransom would be a Council decision and would therefore be made public. As I have previously mentioned, there are many criteria to consider when determining whether ransom should be paid. I will not say any more at this time since we must NOT give any valuable information to those who have attacked us.

Finally, I would like to address the questions of whether this attack could have been avoided and what we will do to improve in the future. I have already mentioned that we are building a new network with all the latest cyber defence enhancements. But the reality is that cyber attackers are incredibly sophisticated and most major corporations, including financial institutions, have been successfully breached. Just in the last month alone, we are aware of many departments in the United States, including the Departments of Defense and the Treasury, that have been successfully attacked. A major US cyber defence company also announced that it had been breached. Many corporations who are successfully attacked never disclose it.

In summary, it is not a question of “if” a corporation will be attacked but rather just a question of “when”. As an interesting side note, in the past three weeks, even with our limited interim network, our IT security systems have identified 13,000 malicious or highly suspicious emails addressed to users of our networks. I would suggest to you that most organizations would have comparable numbers.

That is why we have already shared our lessons learned with many private and public sector organizations and we will continue to do so. We have also shared the details of this incident with both provincial and federal stakeholders. We are developing a robust lessons-learned package that we will share with others so that they can prepare themselves for what lies ahead. And it is worth repeating, that there is no doubt that institutions with which anyone interacts will be breached and sometimes you will not even know about it. Therefore, we continue to recommend that everyone pay close attention to their financials and if anything appears suspicious, please contact your applicable financial institution.

In closing, I would like to thank the entire team that has been working on this incident. Tremendous credit goes to those restoring our networks but equal credit goes to all employees who continue to work through our diminished IT capacity to deliver the services that our community needs.