A few days ago, I made a handling error on an Oracle Linux 7 virtual machine on a freshly configured Exadata X8M. I inadvertently renamed the files “/etc/pam.d/password-auth” and “/etc/pam.d/system-auth”. Result: I was no longer able to connect to the VM in ssh, or even via the virsh console.
I will describe here the steps to regain access by renaming the files with the right name, but it is also applicable if you have lost your root password.
These manipulations have been performed on a virtualized Exadata running KVM (only Exadata X8M uses KVM at the time I’m writing this post). I have not tested them on older Exadata using the XEN hypervisor.
Warning
Before performing these actions, I recommend that you read the entire article once. It is also strongly recommended to make a backup of the concerned VM before starting.
The slightest error could corrupt your virtual machine, so you must remain vigilant at each step.
Context / Prerequisite
In this example, the hostname of the management domain (or dbnode or dom0) is “dbnode1”.
The virtual machine is called “oelvm01.local”.
You need to connect to the dom0 with the root user.
How-to
The first steps consist in mounting the image of the virtual machine as a filesystem on the dom0. To carry out this operation, the VM must imperatively be stopped!
We stop the virtual machine from the management domain:
[root@dbnode1 ~]# vm_maker --stop-domain oelvm01.local
And we use kpartx in order to scan the virtual machine “system” image and detect the partitions in it.
The path of the image must be adapted according to the name of the VM:
[root@dbnode1 ~]# kpartx -a /EXAVMIMAGES/GuestImages/oelvm01.local/System.img
Once done, we can mount the partition on the dom0 (you may need to create the directory “/mnt/imaging/sys” before if it does not exist):
[root@dbnode1 ~]# mount -o loop -t xfs /dev/mapper/loop0p1 /mnt/imaging/sys
At this point, the content of “/mnt/imaging/sys/” corresponds to the content of the “/boot” of the virtual machine. So we can edit the “grub.cfg” file in order to modify the boot options.
We search for the line starting with “linux16” and we add the following options “rd.break enforcing=0“.
They will allow the VM to start whitout the regular root filesystem mounted and whitout SELinux enabled.
In other words: this allow us to get a limited shell on the VM at the next boot without the need to authenticate.
[root@dbnode1 ~]# grep linux16 /mnt/imaging/sys/grub2/grub.cfg
linux16 /vmlinuz-4.14.35-1902.303.5.3.el7uek.x86_64 root=LABEL=DBSYS bootarea=dbsys bootfrom=BOOT boot=LABEL=BOOT fips=0 ro loglevel=7 panic=60 pci=noaer log_buf_len=1m nmi_watchdog=0 transparent_hugepage=never rd_NO_PLYMOUTH audit=1 console=tty1 console=ttyS0,115200n8 crashkernel=448M processor.max_cstate=1 clocksource=tsc nohpet nopmtimer hda=noprobe hdb=noprobe ide0=noprobe pci=nocrs ifnames_skip=100 biosdevname=0 net.ifnames=0 rd.break enforcing=0
Now, we have to unmount the device and start the VM in console mode.
[root@dbnode1 ~]# umount /mnt/imaging/sys
[root@dbnode1 ~]# kpartx -d /EXAVMIMAGES/GuestImages/oelvm01.local/System.img
[root@dbnode1 ~]# virsh start oelvm01.local --console
After a few minutes, the “initramfs switch_root” prompt appears.
The root file system is mounted in read only on “/sysroot”.
We need to remount it in read/write mode in order to be able to edit files or change the root password.
switch_root:/# mount -o remount,rw /sysroot
switch_root:/# mount -o remount,rw /sysroot
Now, depending on your problem you can either change the root password by using the “passwd” command, or modify the problematic files (in my case “/etc/pam.d/password-auth” et “/etc/pam.d/system-auth”).
sh-4.2# cp /etc/pam.d/system-auth.orig /etc/pam.d/system-auth
sh-4.2# cp /etc/pam.d/password-auth.orig /etc/pam.d/password-auth
Once finished, we quit the VM console and we stop the domain.
sh-4.2# exit
switch_root:/# exit
You can exit the console with the following keystroke combination:
- Ctrl + ]
- Ctrl + 5
[root@dbnode1 ~]# vm_maker --stop-domain oelvm01.local
All we have to do now is to redo the first steps in order to modify the grub file and be able to restart our VM in “normal” mode.
[root@dbnode1 ~]# kpartx -a /EXAVMIMAGES/GuestImages/oelvm01.local/System.img
[root@dbnode1 ~]# mount -o loop -t xfs /dev/mapper/loop0p1 /mnt/imaging/sys
[root@dbnode1 ~]# grep linux16 /mnt/imaging/sys/grub2/grub.cfg
linux16 /vmlinuz-4.14.35-1902.303.5.3.el7uek.x86_64 root=LABEL=DBSYS bootarea=dbsys bootfrom=BOOT boot=LABEL=BOOT fips=0 ro loglevel=7 panic=60 pci=noaer log_buf_len=1m nmi_watchdog=0 transparent_hugepage=never rd_NO_PLYMOUTH audit=1 console=tty1 console=ttyS0,115200n8 crashkernel=448M processor.max_cstate=1 clocksource=tsc nohpet nopmtimer hda=noprobe hdb=noprobe ide0=noprobe pci=nocrs ifnames_skip=100 biosdevname=0 net.ifnames=0
[root@dbnode1 ~]# umount /mnt/imaging/sys
[root@dbnode1 ~]# kpartx -d /EXAVMIMAGES/GuestImages/oelvm01.local/System.img
Now we can restart the VM and log in with the root account:
[root@dbnode1 ~]# vm_maker --start-domain oelvm01.local
[root@dbnode1 ~]# ssh root@oelvm01.local
[root@oelvm01 ~]# uname -n
oelvm01.local
[root@oelvm01 ~]# id
uid=0(root) gid=0(root) groups=0(root)
Once again: you have to be really careful when performing these operations. The slightest mistake can be destructive. One of the most important things is to never mount the image of the VM on the dom0 if the VM is started. Also, the VM image must be unmounted using “kpartx -d” before restarting it.
I hope you find this post useful. Stay tuned for more DBA stuff!
