It's been a long time coming but AWS has at last enabled an interactive serial console for de-borking VMs
(2021/04/01)
- Reference: 1617300311
- News link: https://www.theregister.co.uk/2021/04/01/aws_vm_serial_console/
- Source link:
AWS has introduced the "interactive EC2 Serial Console", enabling troubleshooting of virtual machines when normal SSH access is not working, with one [1]user gushing : "I have been waiting 10 years for this moment."
The purpose of serial console access is to enable troubleshooting when an SSH connection is impossible, for example, because of an out-of-memory condition. "It provides a one-click, text-based access to an instances' serial port as though a monitor and keyboard were attached to it," said the [2]AWS post . Previously, admins could see serial console logs, using the command get-console-output, but not enter any commands.
[3]
Back in January 2011, a user reported on the [4]AWS forum (login required) about a case where the console output was "Continue to wait; or Press S to skip mounting or M for manual recovery."
Unfortunately, "there is no way for me to hit 'S'," he said.
[5]
Reasons he gave for requiring the interactive console feature included when boot failed and the SSH daemon did not start, errors configuring the firewall or network which blocked all access, broken networking on the instance, or denial-of-service attacks. This person was building a base instance for a system image, which is the kind of case where fatal errors are more likely.
Admins confronted with an inaccessible EC2 (Elastic Compute Cloud) VM may have another option, which is to stop the instance, detach the storage, mount the storage on a working instance, and edit or recover the files from there. This is not always possible, though.
If the VM uses instance type storage, this cannot be detached. It also requires interruption of service. "I had a customer once that erased their SSH keys, and had a running database cluster on EC2 that they couldn't get access to anymore. That was... fun," [6]said a user on Hacker News, looking forward to the new feature.
[7]
Setting permissions to enable interactive serial console access
If a VM uses ephemeral storage that is not designed for persistent data, why troubleshoot a VM rather than simply deleting it and creating a new one? There may still be good reasons such as analysing the fault or recovering more quickly.
"I used to work on GCE [Google Compute Engine]," [8]said another user, making the point that if a VM has a faulty image where an out-of-memory condition is killing the SSD daemon: "If you replace your instance with another one, you just get another OOM kill."
Playing catch-up
AWS is late in providing this feature. Microsoft Azure has a [9]Serial Console for VMs. Google Cloud Platform has an [10]interactive serial console , and many smaller hosting providers offer it.
Several restrictions apply to the interactive serial console, the most severe being that instances must use the [11]Nitro system , a combination of network, hypervisor, and security hardware which AWS has adopted for many but not all of its EC2 instance types. Second, AWS users do not have permission for the interactive serial console by default. Third, once connected to an interactive serial console, the user with which you log in must have a password. This is often not the case by default with AWS instances, where key pairs may be used instead. Setting this will not be possible when troubleshooting an otherwise inaccessible VM so must be done in advance.
The interactive serial console is also available for Windows instances, where it enabled access to the Special Administration Console (SAC), part of the Emergency Management Services (EMS) tools. These have to be enabled on the Windows instance in advance. If enabled, admins can get access to a range of troubleshooting commands and PowerShell.
[12]
Most AWS users will never need this feature. SSH access does not often fail, and the range of use cases is relatively narrow. But the warm welcome from those who do need it makes it surprising that it has taken so long to implement, and a shame that it is restricted to Nitro instance types. ®
Get our [13]Tech Resources
[1] https://forums.aws.amazon.com/thread.jspa?messageID=979374
[2] https://aws.amazon.com/about-aws/whats-new/2021/03/introducing-ec2-serial-console/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/virtualization&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YGZCen9i3rojIhxEoF@UKAAAAA4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://forums.aws.amazon.com/thread.jspa?messageID=979374
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/virtualization&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YGZCen9i3rojIhxEoF@UKAAAAA4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://news.ycombinator.com/item?id=26640824
[7] https://regmedia.co.uk/2021/04/01/access.png
[8] https://news.ycombinator.com/item?id=26640453
[9] https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-overview
[10] https://cloud.google.com/compute/docs/instances/interacting-with-serial-console
[11] https://aws.amazon.com/ec2/nitro/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/virtualization&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YGZCen9i3rojIhxEoF@UKAAAAA4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://whitepapers.theregister.com/
The purpose of serial console access is to enable troubleshooting when an SSH connection is impossible, for example, because of an out-of-memory condition. "It provides a one-click, text-based access to an instances' serial port as though a monitor and keyboard were attached to it," said the [2]AWS post . Previously, admins could see serial console logs, using the command get-console-output, but not enter any commands.
[3]
Back in January 2011, a user reported on the [4]AWS forum (login required) about a case where the console output was "Continue to wait; or Press S to skip mounting or M for manual recovery."
Unfortunately, "there is no way for me to hit 'S'," he said.
[5]
Reasons he gave for requiring the interactive console feature included when boot failed and the SSH daemon did not start, errors configuring the firewall or network which blocked all access, broken networking on the instance, or denial-of-service attacks. This person was building a base instance for a system image, which is the kind of case where fatal errors are more likely.
Admins confronted with an inaccessible EC2 (Elastic Compute Cloud) VM may have another option, which is to stop the instance, detach the storage, mount the storage on a working instance, and edit or recover the files from there. This is not always possible, though.
If the VM uses instance type storage, this cannot be detached. It also requires interruption of service. "I had a customer once that erased their SSH keys, and had a running database cluster on EC2 that they couldn't get access to anymore. That was... fun," [6]said a user on Hacker News, looking forward to the new feature.
[7]
Setting permissions to enable interactive serial console access
If a VM uses ephemeral storage that is not designed for persistent data, why troubleshoot a VM rather than simply deleting it and creating a new one? There may still be good reasons such as analysing the fault or recovering more quickly.
"I used to work on GCE [Google Compute Engine]," [8]said another user, making the point that if a VM has a faulty image where an out-of-memory condition is killing the SSD daemon: "If you replace your instance with another one, you just get another OOM kill."
Playing catch-up
AWS is late in providing this feature. Microsoft Azure has a [9]Serial Console for VMs. Google Cloud Platform has an [10]interactive serial console , and many smaller hosting providers offer it.
Several restrictions apply to the interactive serial console, the most severe being that instances must use the [11]Nitro system , a combination of network, hypervisor, and security hardware which AWS has adopted for many but not all of its EC2 instance types. Second, AWS users do not have permission for the interactive serial console by default. Third, once connected to an interactive serial console, the user with which you log in must have a password. This is often not the case by default with AWS instances, where key pairs may be used instead. Setting this will not be possible when troubleshooting an otherwise inaccessible VM so must be done in advance.
The interactive serial console is also available for Windows instances, where it enabled access to the Special Administration Console (SAC), part of the Emergency Management Services (EMS) tools. These have to be enabled on the Windows instance in advance. If enabled, admins can get access to a range of troubleshooting commands and PowerShell.
[12]
Most AWS users will never need this feature. SSH access does not often fail, and the range of use cases is relatively narrow. But the warm welcome from those who do need it makes it surprising that it has taken so long to implement, and a shame that it is restricted to Nitro instance types. ®
Get our [13]Tech Resources
[1] https://forums.aws.amazon.com/thread.jspa?messageID=979374
[2] https://aws.amazon.com/about-aws/whats-new/2021/03/introducing-ec2-serial-console/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/virtualization&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YGZCen9i3rojIhxEoF@UKAAAAA4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://forums.aws.amazon.com/thread.jspa?messageID=979374
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/virtualization&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YGZCen9i3rojIhxEoF@UKAAAAA4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://news.ycombinator.com/item?id=26640824
[7] https://regmedia.co.uk/2021/04/01/access.png
[8] https://news.ycombinator.com/item?id=26640453
[9] https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-overview
[10] https://cloud.google.com/compute/docs/instances/interacting-with-serial-console
[11] https://aws.amazon.com/ec2/nitro/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/virtualization&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YGZCen9i3rojIhxEoF@UKAAAAA4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://whitepapers.theregister.com/
Anonymous Coward
The general intention with public cloud infrastructure is that infrastructure should be immutable; if your instances are individually valuable enough you need to log onto them and tenderly nurse them back to health via serial port rather than just blasting them and rebuilding they probably shouldn't be on AWS. Cattle, not pets.
Yay
Kevin McMurtrie
This is probably because I was asking for so many panic logs when attempting to use NVIDIA GPUs in production. (Current theory is that swapping + NVIDIA is broken in EC2)
Still seems a bit sub-par.
DarkwavePunk
I've had to do the AWS EBS shuffle in the past and it's not fun in the slightest. This is welcome in a way although the hoops to get it working look a bit of a ballache as well.
I guess we're all meant to be using a Kubernetes cluster with a DevOps CI pipeline in this brave new world. However, sometimes I'd just like a server to behave like a server whether it be self-hosted tin/VM or in the "cloud".
I am not a AWS user but was surprised to hear that a serial console was only just implemented on there now, the cheap £40 a year VPS I use for hosting a website have this and can be used without needing to open support tickets. And indeed I did need it once when i screwed up the IPTABLES settings and could no longer SSH into it.