News: 1615204007

Covid Logfile Brianna Haley was given one day to be ready to roll out Zoom for 13,000 users at over 1,000 sites.

Haley* is a project analyst for a large healthcare provider that, as COVID-19 marched across the world in March 2020, realised imminent lockdowns meant it would soon be unable to consult with patients.

[1]

And no consultations meant no revenue.

"I got called into a meeting at 7:30 or 8:30 on Monday morning and was told we had to get Zoom done by tomorrow," Haley recalls.

[2]

Haley's company already had Webex licences, but the tool had recently been unreliable and wasn't well used or well loved.

Management wanted Zoom. Now.

"At this point we had zero experience with Zoom and we were talking about adding 2,000 users the first day and up to 13,000 enrolled to use it by the end of the week," Haley said. Which was no small feat given the scale of Haley's employer and the fact it operates several business groups, with a typically diverse networking equipment fleet.

Pre-planning was not an option.

We got a licence to blow up a legacy system. That would not have happened without COVID

"We signed the contract that morning, went to legal, got access to the account at 9:00am, then the team got on a call until 11:00pm that night," Haley said. "We spent 13 hours looking at the configuration guides, guessing the things we had to do to get set up.

"In an organisation like ours a design and acceptance project on this scale might take a year.

"We got them compressed into a day."

On the other side of the world, Chris Moriarty also had a busy day after he was given four hours to move his 250-person business into a new office and get it ready to work from home.

The day started well for Moriarty and Flat Planet, the business process outsourcer he owns and runs in Manila, capital city of the Philippines. Work on the company's new office had finished ahead of schedule and while the fortnight of overlapping leases would be busy with all the minutiae and mess of a move, the old and new offices were just 150m apart so the job promised to be orderly.

But on this day, 16 March 2020, Philippines President Rodrigo Duterte ordered four weeks of "enhanced community quarantine" that would prohibit residents leaving home for 28 days other than to buy food or seek medical attention. An accompanying curfew came into effect at sundown.

Four hours away.

"I called in my wife, maids, the builders from the new office," Moriarty recalled. "We were all just sprinting up and down the street. We moved the server cupboard, the filing cabinets, hundreds of PCs. We did it in four hours, without any trucks or anything."

Other businesses had more time to prepare. Global software giant VMware operates offices in China so news of lockdowns there saw the company's global risk management committee begin to plan for many possibilities in the week of 20 January 2020.

Cisco rations VPNs for staff as strain of 100,000+ home workers hits its network [3]READ MORE

But a week later the company was still conducting business as usual. That's when Jason Conyard, now VMware's CIO but then vice-president for IT, Colleague Experience and Technology, got on a plane to visit the company's service desk team in Bangalore, India.

"I was packing my bag and had the TV on, and I asked myself if it was a good idea to travel," Conyard said. He decided the trip was safe, but by the time he made it to Bangalore and reconnected with the news, his thinking changed.

"It felt unusual and different," he said. "And it prompted me to start having some more pointed conversations with my team. I was still looking through a regional lens, asking what equipment do we have? What vendors do we have dependencies on? Are we ready for lockdowns?"

Conyard's trip to Bangalore turned out to be well timed because it coincided with China's lockdowns affecting VMware staff.

"We were having conversations about the need to be empathetic with colleagues in China," Conyard told The Register . "We started to set the tone for the rest of the world."

The picture wasn't as clear for Australian software upstart Atlassian, which started asking similar questions in the first week of March 2020 when the company's head of workplace and technology, Ross Chippendale, returned to work after a break. "I started talking to the team about things like capacity and security if we needed to work from home for a few weeks," he said.

By this time, the company's CIO was also thinking about how to handle a possible pandemic and chatted to Chippendale about Atlassian's options.

As a cloudy company that encouraged working from home, Atlassian had already invested in remote work capabilities.

More than one-fifth of smartphone sales evaporate in China as pandemic grips Middle Kingdom [4]READ MORE

"In theory we had capacity for 5,000-plus remote workers," Chippendale said. "But we had never tested it."

"We believed it would work. When the CIO asked me if it would, I said: 'I'm pretty sure.'"

That experience of having untested remote work infrastructure in place was not unusual.

Bob Fuller*, operations manager at an Australian law firm, was most of the way through a VPN upgrade in March 2020. "We had Azure Active Directory and multi-factor authentication in place," he said. "We were in a really good position for ad hoc remote work."

But not for universal remote work. And the firm hadn't quite considered how to handle the security and confidentiality sensibilities of legal documents on employee-owned PCs and untrusted networks.

In the UK, SaaS accounting software company Crunch had also done some foundational work as part of a disaster recovery plan refresh that had replaced hard-wired phones with a cloudy PBX and implemented Amazon's Workspaces desktop-as-a-service.

But the company had a culture of showing up in the office. An annual work-from-home day was in place to test DR readiness – and those AWS Workspaces – and developers had started agitating for the chance to regularly work from home.

Dell, VMware withdraw fiscal 2021 guidance amid humming global supply chain and remote work boom [5]READ MORE

However, those efforts had only advanced to the point of contemplating tests of having a whole team working remotely on a single day. Florentin Raud and Abdul Rashid, Crunch's head of infrastructure, support and IT, and support technician respectively, had yet to greenlight that test.

Kane Ivers*, director of information security and infrastructure for a US-based law firm that specialises in debt recovery, also had an unproven remote-work tool in the form of a newly upgraded VPN.

All of these IT pros were about to find out if their efforts would be enough to cope with a pandemic.

And all were about to enter a weeks-long blur of insane hours and challenges that would force them to take on and meet new challenges they'd never previously contemplated.

That day in March

Someone at a VPN vendor stuffed up licensing by a factor of 20, and that mistake ended up being a blessing for Kane Ivers.

Ivers can't remember the day he discovered the error but can recall that in mid-March 2020 he was called into an executive team meeting and asked what it would take to get a large portion of the company working from home.

Ivers was able to answer that the company had all it needed to get everyone connected because the rep who sold him a new VPN had assumed it was a model capable of handling 50 concurrent IP addresses. But somewhere along the line that mistake meant the company ended up with a device that could handle 1,000 connections and had licences to match. And all that without blowing budgets!

The VPN was already in use by travelling executives and part-time workers so the company had processes in place for its use. Client software for the company's preferred VPN client was easy to find and easy to install, even for Chromebooks.

In theory we had capacity for 5,000-plus remote workers. But we had never tested it

Others have similar stories of their not-entirely-purposeful preparations working.

At Crunch in the UK, DR tests of AWS Workspaces meant staff were familiar with remote working. When the company needed to scale its virtual desktop fleet, Amazon obliged.

VMware already operated a large private cloud and was able to reallocate resources to deliver more virtual desktops to staff suddenly working remotely.

Bob Fuller in Australia was pleased that his new VPN worked, while his firm's use of Office 365 meant employees could get most of the functionality they used in the office from the web.

But while the back office worked, and clouds scaled, end users floundered.

Sysadmin/logistics expert/home broadband help desk...

"The first day of service was insanity," Ivers said, because users turned out to need more help than he'd expected.

Across all the interviews The Register conducted for this story, the issues were similar. Home internet connections wouldn't work, users couldn't install VPNs or make a connection. Many staff lacked a PC to work with, or missed their dual monitors, or managed to connect to a remote desktop with an iPad and let it be known they weren't going to do that for long.

Videoconferencing flummoxed many. Executives weighed in with worries about the legality of handling sensitive documents at home. Windows updates on user-owned machines slowed things down at the worst moments.

IT departments were expected to solve every problem.

So while Kane Ivers' VPN may been mercifully overprovisioned, it added complexity to the login process.

India suggests cloud storage to slow the spread of coronavirus [6]READ MORE

"Every morning from 7am to 8:30am our team of four would just do help desk tickets to unlock accounts," he said. And then he could get on with the many requests that were best addressed by advising users to move their PC closer to their Wi-Fi routers.

Ivers' employer also settled on RDP into actual PCs in the company's office as its preferred remote access regime so that confidential files never left the building. But remote users working with a PC-inside-a-PC made for more service desk work because they'd often shut down the wrong PC at the end of the day, leaving the IT team to restart the on-prem device. And to brave the journey into the office as the pandemic raged.

And by this time layoffs had shrunk Ivers' team from four to one-a-and-a-half people.

Bob Fuller was also working hard.

"For the first three weeks everyone on the IT team was doing 12-hour days," he recalled. "We became an ISP troubleshooting desk, checking home Wi-Fi, making sure that people weren't streaming video or torrenting while they worked.

"The first thing we did was nearly always a broadband speed test. We did so many speed tests and then we could triage to see if the issue was local, or at our end.

"We quickly learned to be more tolerant of people going through things for the first time. We were stressed. We were tired. We were all figuring out how to do things we had never done before."

Every morning from 7am to 8:30am our team would just do help desk tickets to unlock accounts

So were some staff at VMware, where Jason Conyard realised that the teams responsible for maintaining end-user technology in its offices had the skills to help newly remote workers.

"We added those people to the service desk. And we told everyone to forget about meeting on-hold time targets. Just do what you have to do to help."

But even with the company's global reach and ability to mobilise extra staff, Conyard didn't get a day off from late January until early April.

For Brianna Haley's team, doing whatever it took meant three weeks of working 7am to 8pm. "In the first week I ate all of my meals at my desk," she said.

At Crunch in the UK, the IT team was stretched because the company's SaaS tools include a payroll management application, and worker furloughs generated a lot of payroll-related work. Florentin Raud and Abdul Rashid suddenly had a significant group of users working longer-than-usual hours in a new environment.

They did get one break: many of the Crunch team had gaming PCs at home. Those powerful machines had no trouble whatsoever handling AWS Workspaces or other software.

But the IT team at Crunch and elsewhere also found themselves charged with getting PCs to workers who didn't have a suitable machine at home.

IT departments got that job even as the logistics industry itself was trying to handle increased demand while learning about and complying with the new rules of the pandemic.

Logistics at Atlassian were complicated by the fact that the company kept hiring new staff even as lockdowns descended. So while most of the company's existing workers already had company-issued laptops, the IT team had to keep new devices flowing even as PCs and laptops became hard to find.

Microsoft names priority users for new Azure capacity – emergency services, government, remote workers top the list [7]READ MORE

As a big buyer, Atlassian had willing suppliers in some places. But the situation was highly variable.

"We experienced different levels of lockdown in different parts of the world," Chippendale said, which meant he and his team needed to understand what was possible in different nations.

Daily logistics crisis meetings tackled those issues one by one.

"We addressed things like getting 10 laptops and monitors to northern India," Chippendale explained. But privacy concerns mean the IT department shouldn't know workers' home addresses. That meant bringing in the HR department, which suddenly found itself working alongside IT every day.

In the Philippines, meanwhile, Chris Moriarty was trying to arrange wireless broadband for his team. Doing so meant not just figuring out if deliveries were even possible with curfews and movement restrictions in place, but also learning which carriers offered viable service levels across Manila's sprawling suburbs.

Fancy some post-weekend reading? How's this for a potboiler: The source code for UK, Australia's coronavirus contact-tracing apps [8]READ MORE

Back in India, VMware discovered that many of its employees' homes were built with steel-reinforced concrete, a choice that reduced the effectiveness of wireless broadband. Wired broadband is not Indian carriers' primary product – the nation has just 20 million wired connections but over 700 million wireless users.

Which was just one of the many challenges IT teams faced that usually just required working the phones until they could find a courier, a carrier, or a vendor willing to help – then trying to get approval for the price.

Getting serious

By April or May, IT departments had made working from home viable and stable and were able to turn their attention to also making it elegant and resilient.

For Brianna Haley and the healthcare provider she serves, that meant integrating Zoom with enterprise directories and calendars so users were properly authenticated and could schedule meetings.

"We had to identify who were doctors, who worked on the business, and that can be difficult when you are working across as many companies as we were and going through mergers."

It also meant prioritising traffic.

"We decided that Zoom should only get a certain amount of bandwidth, but then users would ask about that. We'd explain that we need the network available for all other devices, that we don't mind video meetings within the organisation, but we can't have it impacting the arrival of a patient scan into the building."

A design and acceptance project on this scale might take a year. We compressed it into a day

For Kane Ivers it meant understanding the local telecommunications market, in which Comcast is the dominant consumer ISP but CenturyLink provides most business links. "One day the peering between the two was obviously under massive stress," he explained. The result was that remote access became unreliable.

"So we bought a circuit from Comcast to remove the peering problems."

At Crunch, the company realised it would need to come up with a new remote access plan once its first bill for AWS Workspace arrived.

"The bill wasn't terrible. We were expecting it in a way," Raud and Rashid told The Register .

Indeed, AWS had given the company a heads-up about its consumption.

"Workspaces is not a very AWS-like service because it is a bit slower to provision and hand back resources," the pair said. "AWS reached out and said you will reach your capacity limit."

Crunch's tech team wasn't sure if AWS was giving them excellent service, trying capacity management of its own cloud, or a bit of both.

The combined effect of AWS's warnings and that big cloud bill led the pair to decide it was time to move from disaster recovery mode into something more suitable for the long haul. It was at this point that Crunch started to send equipment home, explaining to management that the sunk cost of kit needed to be used rather than shelling out for monthly DaaS bills. Workspaces use dropped from 124 in March, to 100 in April. By October the company was down to 20 workspaces, but had increased its use of an AWS VPN to reach apps and data in the cloud.

We told everyone to forget about targets. Just do what you have to do to help

Crunch also started to deal with local ISPs to make sure staff had reliable connections.

"We reached out to an ISP that we trust, explained our needs and asked if they have any deals." The ISP was happy to help, staff liked the offers, and Crunch liked the results so much it eventually assembled a panel of four ISPs it could recommend to its team.

Back in Australia, some courts had started to conduct hearings online with Microsoft Teams with third-party videoconferencing hardware. Skype for Business had become a mainstay for Bob Fuller's employer. He now got to enable dial in to Teams, and make sure the firm's Teams rig could work with courts' videoconferences.

Doing so meant integration with the firm's Cisco meeting room hardware and a limited return to staff working in the company's offices.

"We did social distancing in the meeting rooms," he said.

Also down under, Atlassian realised that its service desk had gone virtual, and that its licences for remote PC maintenance tool Jamf would come in more than handy. Along the way, Ross Chippendale said Atlassian realised it had not explored Jamf's full potential. As mid-year passed, Atlassian started to bring new rigour to its remote operations, exercising greater control over the software installed on its employees' machines.

A new view

COVID helped Atlassian change quickly, Chippendale said.

"Ticket volumes have not changed," he said, "but the nature of the interactions we have is different because they are on Slack, not at a desk. We've changed by automating a bunch of tasks and the time to resolve things has gone down.

"It's led to a change in the way we deploy the workforce."

Atlassian thinks it can also productise some of its lockdown experience.

"We are looking at IT service management (ITSM) and our Jira service desk product is looking to explode into that space," Chippendale said. "I have often said we are quite a useful test case. My team has a good relationship with the product teams."

At Crunch, the pandemic has meant big change as the necessity to enable work-from-home has seen staff start to move to areas that are more affordable but located further away from the office.

UK finds itself almost alone with centralized virus contact-tracing app that probably won't work well, asks for your location, may be illegal [9]READ MORE

With Crunch HQ located in pricey Brighton, the chance to move to other parts of the UK is appreciated by staff.

"In London a shoebox costs thousands a month to rent, and Brighton is close to that," says Crunch's Rashid. "The attitude has become that if you are going to be in our house all day, you might as well move ten miles away and get a house for the same price as that shoebox."

As staff moved so did attitudes in Crunch's IT team.

"I think what we learned is to trust users to tell us if anything is wrong," Rashid said, but the tech team must communicate more about potential outages. "In the office we can pick up the cues from users. Now it is more on us to say if something is not working."

For Brianna Haley, the pandemic has shaken up IT culture.

"We got a licence to blow up a legacy system," she said. "Would I have wanted to do that without COVID? Maybe not. But we are now definitely pushing into new ways of providing care, leveraging Zoom even in clinical situations. That would not have happened without COVID." ®

The Register is planning a series of [10]Covid Logfile stories that, hopefully, offer a first draft of 2020 history from the perspective of the IT community. If you'd like your experience to be considered, we're looking for:

Stories of being laid off due to the pandemic, and the process of finding a new gig during an economic crash

Experiences of adopting collaboration tools, from both technical and cultural viewpoints

Home office setups and how they evolved. The odder the better!

To share your story, mail us [11]here .

[12]

Some names in this article are pseudonyms, as interviewees were not granted permission by their employers to speak with The Register.

Get our [13]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YEZYL9LtxQmd2ImhwJdPzgAAAA8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YEZYL9LtxQmd2ImhwJdPzgAAAA8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[3] https://www.theregister.com/2020/04/02/cisco_rations_staff_vpn/

[4] https://www.theregister.com/2020/04/30/while_chinas_smartphone_market_contracts/

[5] https://www.theregister.com/2020/03/27/dell_withdraws_fy_2021_fiscal/

[6] https://www.theregister.com/2020/03/31/india_digilocker_promotion/

[7] https://www.theregister.com/2020/03/22/microsoft_azure_health_government_priority/

[8] https://www.theregister.com/2020/05/09/coronavirus_tracing_app_source_code/

[9] https://www.theregister.com/2020/05/05/uk_coronavirus_app/

[10] https://www.theregister.com/Tag/covid-logfile

[11] mailto:simon.sharwood@sitpub.com

[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YEZYL9LtxQmd2ImhwJdPzgAAAA8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[13] https://whitepapers.theregister.com/