# Remote attestation of SEV-SNP confidential VMs using e-vTPMs

Vikram Narayanan  University of Utah, Salt Lake City, UT, USA Claudio Carvalho
IBM Research, Yorktown Heights, NY, USA  Angelo Ruocco  IBM Research, Yorktown
Heights, NY, USA  Gheorghe Almási  IBM Research, Yorktown Heights, NY, USA
James Bottomley  IBM Research, Yorktown Heights, NY, USA  Mengmei Ye  IBM
Research, Yorktown Heights, NY, USA  Tobin Feldman-Fitzthum  IBM Research,
Yorktown Heights, NY, USA  Daniele Buono  IBM Research, Yorktown Heights, NY,
USA  Hubertus Franke  IBM Research, Yorktown Heights, NY, USA  Anton Burtsev
University of Utah, Salt Lake City, UT, USA

## Abstract

Departing from your data is safe with us model where the cloud infrastructure
is trusted, cloud tenants are shifting towards a model in which the cloud
provider is not part of the trust domain. Both silicon and cloud vendors are
trying to address this shift by introducing confidential computing – an
umbrella term that provides mechanisms for protecting the data in-use through
encryption below the hardware boundary of the CPU, e.g., Intel Software Guard
Extensions (SGX), AMD secure encrypted virtualization (SEV), Intel trust
domain extensions (TDX), etc.

In this work, we design and implement a virtual trusted platform module (vTPM)
that virtualizes the hardware root-of-trust without requiring to trust the
cloud provider. To ensure the security of a vTPM in a provider-controlled
environment, we leverage unique isolation properties of the SEV-SNP hardware
and a novel approach to ephemeral TPM state management. Specifically, we
develop a stateless _ephemeral_ vTPM that supports remote attestation without
persistent state. This allows us to pair each confidential VM with a private
instance of a vTPM that is completely isolated from the provider-controlled
environment and other VMs. We built our prototype entirely on open-source
components – Qemu, Linux, and Keylime. Though our work is AMD-specific, a
similar approach could be used to build remote attestation protocol on other
trusted execution environments (TEE).

##  1 Introduction

Over the last two decades, public clouds have become de facto standard
execution environments for deploying a broad range of modern software. The
move to the cloud, however, created a unique security challenge – cloud
tenants no longer own or control an environment in which their software is
deployed. Tenants are required to trust not only the provider itself, but also
a complex software stack virtualizing the physical hardware across multiple
users. In the last decade three most widely deployed virtual machine monitors
(VMMs) – Xen, KVM, and VMware – suffered from 428 [30], 111 [11] and 154 [28]
vulnerabilities each. Moreover, physical access to the system opens the door
for a range of hardware attacks, e.g., memory extraction such as cold-boot
[76], RAMBleed [53, 70], etc.

Recent CPU architectures have introduced support for a hardware-protected
trusted execution environments (TEEs) as a way to minimize the trusted
computing base (TCB) of a cloud application [41, 2, 8, 9, 48]. By using a TEE,
the cloud provider infrastructure (including the hypervisor) is removed from
the TCB, since the application is isolated from it.

One of such TEE solutions is SEV-SNP, a variant of AMD secure encrypted
virtualization (SEV) technology available in the EPYC 7003 series of
processors [4]. SEV-SNP launches a guest virtual machine in a hardware context
that is isolated from the rest of the system by means of memory encryption.
Hence, even an attacker who has full access to both hardware and privileged
cloud software (i.e., platform firmware, hypervisor and operating system)
cannot access the encrypted state of the protected guest VM (also referred as
confidential VM).

Thus hardware memory encryption provides confidentiality of the application’s
code and data. Unfortunately, confidentiality without integrity does not
provide strong security guarantees. A range of attacks allow a malicious
entity on the host to attack the execution of the confidential VM by modifying
the code, data, or configuration parameters during the boot sequence or later
as long as attacker posesses administrative privileges (e.g., enabling debug
flags that could leak secrets, loading malicious kernel extensions,
downgrading security-critical subsystems to vulnerable versions, etc.)

To ensure integrity, i.e., the guarantee that the system is not tampered with,
modern systems rely on a combination of _measured boot_ [77, 61] and _runtime
attestation_ [38, 47]. Measured boot consists of measuring all boot time
binaries, i.e., platform firmware, bootloader(s), and the operating system.
Runtime attestation combines measured boot with integrity measurement
architecture (IMA) that uses the same principle – measuring all executables
loaded into the memory – after the system is booted. Attestation works by
comparing entries in the measured boot and IMA logs with a pre-defined set of
acceptable values (called an _attestation policy_) and exposing any
measurements that do not conform to policy expectations.

Support for attestation requires a _root-of-trust device_ , i.e., an integrity
protected location that can store measurements in a trustworthy manner, extend
them, and authenticate the measurement logs to the user (remote attestation).
This device is typically a trusted platform module (TPM) chip. TEE
technologies typically offer mechanisms to support _pre-attestation_ , i.e.,
measuring all the memory initialized in the confidential VM before boot, but
lack mechanisms that allow the software stack to extend and log subsequent
measurements.

Thus, to perform continuous runtime attestation and integrity monitoring of a
confidential VM, the VM needs a different root-of-trust. Given the existing
software stack for TPMs, it is natural to think of a virtual TPM (vTPM).
Providing vTPMs with strong security guarantees in an environment in which
neither the guest system nor provider are trusted is challenging. Several
design constraints are critical for the security of the system:

  * •

Isolation: To ensure security of an encrypted VM, the vTPM must be protected
from the host controlled by the provider (an obvious TEE requirement). In
addition, the vTPM also needs isolation from the guest operating system, since
it acts as a root-of-trust device for attestation. Existing vTPM designs trust
the host OS and or the hypervisor to run a vTPM manager and vTPM instances
[26, 13, 6, 1], isolating the vTPM from the guest but leaving it open to
attack from the provider side.

  * •

Secure communication: In a typical vTPM, the TPM commands and responses are
transmitted through the untrusted hypervisor [34, 49, 57, 66, 32]. An attacker
can interpose on the channel and alter the request or response defeating the
security guarantees offered by a TPM [36]. To provide confidentiality and
integrity of messages exchanged between the VM and the vTPM in an untrusted
host environment, we require a secure communication channel.

  * •

Persistent state: A physical TPM guards its internal state inside a mechanical
chip wrapper and as such, the TPM state cannot be easily exfiltrated.

By contrast, a vTPM instance requires the injection of its state every time it
is created. The persistent state needs to be properly secured both at rest,
and while in use. At rest protection can be done by using encryption
mechanisms on the permanent state file. In use protection is more difficult,
since it requires both protecting the vTPM memory from improper access, and
ensuring that the state cannot be exfiltrated with legitimate operations
coming from a malicious actor.

In this work, we propose SVSM-vTPM, a new vTPM architecture that solves the
above security challenges. Our work leverages unique architectural properties
of the AMD SEV-SNP execution environment; however, we will discuss how to
generalize this solution at the end of the paper. Specifically, we rely on the
VM privilege level (VMPL) feature offered by the secure VM service module
specification (SVSM) to isolate the vTPM from both the host and the guest
system. With VMPLs, we implement a vTPM running inside a privilege isolated
memory region inside the guest VM address space. Since the vTPM is still
running inside the confidential VM, it is also isolated from the host.
Additionally, being inside the confidential VM allows us to automatically
protect the request and response messages to and from the TPM by leveraging
the confidential VM memory encryption. This allows us, by construction, to
eliminate the need for establishing a secure communication channel between the
VM and the vTPM.

Finally, we address the problems of persistent vTPM state by proposing a
stateless, ephemeral vTPM. Specifically, we pair each confidential VM with a
private instance of a vTPM that manufactures fresh state and keys every time
it boots. This means that there is no shared state managed by a TPM manager
and no persistent vTPM state to protect. The vTPM state lives entirely within
the encrypted memory of the virtual machine, with no mechanism for exporting
it, allowing us to address a whole class of attacks based on exfiltration of
vTPM state. The cost of doing this ephemeral vTPM is that the TPM itself can
retain no persistent keys or non-volatile indexes. While this makes our vTPM
implementation not compliant to the specifications, we note that persistent
state is not required for the attestation and integrity monitoring procedures
we describe in this paper.

Our contributions are as follows:

  * •

We propose the use of an ephemeral vTPM to remove attacks to the vTPM state.

  * •

We are the first to leverage the new features of AMD SEV to provide a secure
implementation of a vTPM.

  * •

We demonstrate a full remote attestation workflow for our SVSM-vTPM solution,
implicitly proving that remote attestation frameworks can provide measured
boot and remote attestation with an ephemeral vTPM.

##  2 Background

###  2.1 AMD secure encrypted virtualization

In 2016, AMD introduced secure encrypted virtualization (SEV) where the entire
virtual machine is encrypted with an ephemeral key that is managed by a
dedicated co-processor, AMD secure processor (AMD-SP). AMD-SP takes care of
lifecycle management of the SEV VMs [31] and serves as the integrated root-of-
trust for the AMD processor [55]. By using a unique key per VM, SEV isolates
the guest VMs from the rest of the host operating system and from other
guests.

Since 2016, AMD incrementally added additional protection features to SEV. AMD
introduced SEV-ES (SEV encrypted state) to protect the register state in the
virtual machine control block (VMCB) with encryption and integrity protection
[17]. To communicate and share data with the hypervisor during hypercalls, a
new structure called guest hypervisor communication block (GHCB) was
introduced [21] that would remain unencrypted. In their next version, SEV-SNP
(secure nested paging), AMD introduced a reverse mapping table (RMP) which
performs page validation and keeps track of page ownership to prevent replay
attacks [4].

#### Virtual machine privilege levels

AMD also introduced virtual machine privilege levels (VMPLs) in SEV-SNP.
Similar to protection rings in x86 architecture, VMPLs allow a guest VM
address space to be subdivided into four levels with different privileges
(with VMPL0 being the highest privilege level). These levels can be used to
implement privilege isolated abstraction layers within a confidential guest
virtual machine [4].

The introduction of VMPL allows the design and deployment of secure services
that are completely isolated from the untrusted host operating system and the
guest VM. Secure VM service module (SVSM) specification [19] also defines a
standard interface for communicating between various services offered by the
software running at VMPL0 and the guest operating system. The protocol uses
registers to pass the arguments and return values. In the absence of SVSM
firmware, the entire guest VM can execute under VMPL0 unmodified. However,
with SVSM, they run at a lower privilege level, corresponding to a higher VMPL
(i.e., 1-3), and require interaction with the SVSM for some privileged
operations.

###  2.2 Virtual trusted platform module (vTPM)

A vTPM is a pure software implementation of a TPM module as defined by the TPM
2.0 specification [24]. vTPM enables virtualization of a hardware root of
trust across multiple entities, i..e, virtual machines, and is aimed at
providing functionality identical to a hardware TPM. Berger et al. [34]
proposed the first design for virtualizing a TPM that can be used for
providing TPM functionalities to virtual machines. Their design consists of a
vTPM manager and a set of vTPM instances, where the vTPM manager executes as
part of the VMM and takes care of multiplexing physical hardware across
multiple VMs. Berger et al. extend the TPM command specification to include
support for creating virtual instances and rely on hardware TPM for
establishing trust.

Stumpf et al. [68] proposed a virtual TPM design by applying hardware
virtualization techniques from Intel VT-x technology. Their multi-context TPM
contains different modes of execution and has a dedicated TPM control
structure (TPMCS) for every VM, which would be loaded by the VMM before
invoking the TPM commands. Several vTPM architectures were proposed over the
years: from a generalized vTPM [66] to separating vTPM functionalities across
Xen domains with different privileges [32, 49, 57]. However, they were either
placing trust on the host environment (VMM, host OS) or relying on hardware
TPM for establishing trust. Unfortunately, none of those designs satisfies the
security and confidentiality requirements of confidential computing. Recent
vTPM designs move their implementation inside a TEE such as Intel SGX [69, 75,
74, 62]. Though this design offers protection from the cloud provider, the
state of the TPM must be securely stored and should be protected against
rollback attacks.

###  2.3 Runtime integrity

Modern kernels deploy a number of security mechanisms to prevent runtime
exploitation of low-level vulnerabilities [37], e.g., stack canaries [42],
address space randomization (ASLR) [67], data execution prevention (DEP) [73],
superuser-mode execution and access prevention [43, 39], and even control-flow
[40] and code-pointer integrity [52]. These mechanisms, however, are not
designed to stop an attacker who has control over the boot sequence of a
system, e.g., an untrusted cloud provider, or gains administrative privileges
and can load malicious kernel extensions, or downgrade security critical
subsystems to exploitable versions. To prevent such attacks modern systems
rely on a combination of measured boot and runtime integrity monitoring.

Measured boot is the process of recording the measurements of all boot
components during the system initialization process. The hash of the
components are recorded in a log file that is authenticated using the Trusted
Platform Module (TPM). This authentication works by extending TPM Platform
Configuration Registers with digests of individual events in the boot log. A
TPM signed _quote_ an be used to vouch for the accuracy of the log.

Integrity measurement architecture (IMA) is a Linux subsytem that collects the
hashes of files when opened, before it is read or executed [65]. To ensure the
integrity of these measurements, they are extended into the TPM PCRs, similar
to the measured boot log. Together with measured boot, IMA enables remote
attestation to ensure the runtime integrity of the system.

Remote attestation is a process that proves or ascertains certain properties
of a set of devices/machines to an outside observer. As an example, one might
be interested to ascertain the booted kernel, on a set of machines in a data
center. These properties of interest are cumulatively called an attestation
policy.

##  3 Threat model

We assume that an attacker has physical access to the machine and unrestricted
privileges on the software and firmware executing on the host machine, i.e.,
firmware, hypervisor and virtualization stack, and the host operating system.
However, the memory of the confidential VM is protected by the AMD SEV
technology, i.e., encrypted with a key only known to the AMD secure processor
(AMD-SP). We trust the AMD hardware and the implementation of SEV-SNP and
SVSM.

Ciphertext side channel attacks [56, 54] on the SEV encrypted VM (by building
a dictionary of plaintext-ciphertext pairs) are out of scope. Attacks against
the integrity measurement architecture (IMA) such as TOCTOU [35], other
measurement gaps such as code injected by extended berkeley packet filter
(eBPF) are out of scope. Also, runtime attacks exploiting stack or heap
overflows such as return-oriented programming on the guest VM are out of scope
as IMA measures only the persistent files.

Figure 1: SVSM-vTPM architecture and its components

##  4 TPM virtualization with SVSM

SVSM-vTPM is a secure virtual TPM designed to enable remote attestation and
runtime integrity measurement in a provider-controlled confidential computing
environment backed by an AMD SEV hardware. Specifically, we do not trust any
software on the host machine. To achieve strong isolation from the host, we
leverage unique capabilities of AMD SEV environment and execute a virtual TPM
instance along with the guest system inside a hardware-protected TEE enclave
(Figure 1). The entire SEV-SNP confidential VM memory is encrypted by the AMD-
SP. SVSM-vTPM runs inside the VM privilege level 0 (VMPL0), which allows us to
both isolate it from the rest of the guest system and provide secure
communication between the guest and the TPM. Specifically, we load a minimal
baremetal execution environment in VMPL0 when a new confidential VM is
created. Finally, we completely eliminate the burden of TPM state management
such as preserving the state, injecting it to the correct confidential VM
during boot-up, and also prevent a whole class of attacks based on
exfiltration of the TPM state with a novel idea of an ephemeral TPM – our TPM
instances have no persistent to save or guard against.

###  4.1 Isolation

As vTPM offers a virtual root-of-trust for the virtual machine, it has to be
hosted in an environment that provides strong isolation of its state and is
designed to minimize the attack surface for a potential attacker. Arguably,
two design flaws undermine the security of existing vTPMs to be used in a
confidential computing environment.

First, until recently, the cloud provider was a de facto part of the trust
domain. vTPMs were often managed and implemented as a component inside the
hypervisor [34] or as a part of the virtualization stack [49, 57, 32]. To
reduce the attack surface on the component hosting the vTPM, several
alternative vTPM architectures were proposed. Triglav vTPM utilized dynamic
root of trust (DRTM) as a mechanism to ensure the integrity of the hypervisor
[60]. Another vTPM solution utilized x86 system management mode (SMM) for
isolation and protection of the TPM [58]. Though such designs offer some form
of protection against a non-malicious cloud environment, they do not satisfy
the requirements of confidential computing where the entire host environment
is untrusted. Recent TEE-based vTPMs run the vTPM manager and several
instances in a hardware isolated TEE such as SGX [69, 74, 75], AMD SEV
confidential VM [62] or in ARM Trustzone [63].

Second, historically, virtualization of TPM relied on a centralized
architecture. The core part of the vTPM, a vTPM manager, responsible for
instantiating a TPM, multiplexing the communication between multiple VMs and
vTPMs, and saving the TPM state in a secure location was shared across all
vTPM instances [34, 32, 49, 57, 62]. As the manager handles the lifecycle of
all vTPMs on a machine and has access to the physical TPM hardware, it
naturally becomes a central point for attack. A malicious VM can launch
attacks ranging from a simple denial-of-service to sophisticated attacks
trying to exfiltrate the secrets by exploiting the vulnerabilities in a
centralized the software component of the vTPM manager. If exploited, the
security of all the vTPMs handled by the manager is compromised.

Figure 2: Generating SEV-SNP attestation report inside SVSM-vTPM

#### Private, isolated TPMs

Instead of relying on a central vTPM manager that manages several instances of
vTPM in an untrusted environment, we base our design on two insights. First,
to provide strong isolation of the vTPM code, we leverage the architectural
support offered by AMD SEV. Second, to avoid centralized management, we rely
on SVSM specification that offers a way to implement secure services inside
the guest VM.

Specifically, to ensure isolation, we leverage the VM privilege levels inside
the confidential VM address space provided by the SVSM specification as part
of the SEV-SNP architecture. In our architecture, every confidential VM has
its own private vTPM that runs at a higher privilege level (i.e., VMPL0)
inside each confidential VM and is encrypted by AMD-SP and has the same
isolation guarantees of an encrypted VM.

By running our vTPM within an isolated privilege level within the guest
address space, we eliminate all the attacks that could be mounted on the
component that runs the vTPM. Additionally, operating at the VMPL0 offers
additional protection that it cannot be interfered by the guest or the host
OS.

We use Qemu/KVM environment for running the confidential VM. Figure 2 shows
how a confidential VM is launched. A user provides the boot-time binaries
(typically SVSM and OVMF) to be loaded as part of the guest image (
\raisebox{-0.9pt}{1}⃝ ). Qemu communicates with the KVM which communicates
with the SEV firmware running inside the AMD-SP through an API interface to
create a confidential VM ( \raisebox{-0.9pt}{2}⃝ ). The SVSM firmware is
placed in VMPL0 and the OVMF firmware and the rest of the guest environment
(i.e., the kernel and initrd in case of direct boot) is placed at VMPL1.

Unlike a regular programming environment that provides operating system
abstractions (e.g., syscalls, timers, etc) and feature-rich libraries, SVSM
firmware runs on a restrictive bare-metal environment without access to such
features. Enclave environments often come with such restrictions, for
instance, one would need a sophisticated library OS [71] to run unmodified
applications inside SGX. However, in a bare-metal environment such as SVSM,
one do no have operating system abstractions such as timers/clocks,
availability of crypto libraries, etc. However, a vTPM needs to have access to
timers, random numbers, and cryptographic libraries for realizing a software
TPM module. We manually port the necessary libraries to satisfy the
dependencies of the TPM module. Due to the encrypted code pages and the lack
of interfaces between the debugger and Qemu to install breakpoints inside the
encrypted pages, we had to rely on print statements during development for
debugging.

###  4.2 Secure communication between VM and vTPM

The communication channel between a VM and a corresponding vTPM is a potential
target for a range of security attacks, e.g., by altering the TPM command
requests and response buffers it is possible to subvert measured boot and
runtime attestation protocols [36]. One way to mitigate such attacks is to
secure the communication channel by implementing standards such as TPM HMAC
and encryption [24] or DMTF secure protocol and data model (SPDM)
specification [20]. Though the TPM specification describes encryption and HMAC
security layers, there are very few TPM implementations that support them.
Developing a complex secure communication protocol such as SPDM requires a
large engineering effort. Recent vTPM designs that rely on a hardware-
protected TEEs implement secure communication channel using transport layer
security (TLS) protocol [16]. Unfortunately, even a standard TLS protocol
negatively affects the TCB size of the TPM.

#### Secure communication

Instead of implementing a secure communication protocol, we rely on the
mechanism provided by AMD SEV and its ability to pass execution between
virtual machine privilege levels. While the transition between VMPL1 and VMPL0
triggers an exit into an untrusted hypervisor controlled by the cloud
provider, the internals of the message remain protected inside the hardware-
encrypted memory. Moreover, AMD SEV specification ensures that the hypervisor
can only resume execution of the VM at a corresponding privilege level, i.e.,
VMPL0, if the guest system triggers a transition to the TPM. Hence, the
hypervisor is unable to suppress messages unless the whole VM is halted.

To implement the communication protocol, we developed a modified version of
the Qemu emulator and the guest Linux kernel to interact with our SVSM-vTPM
operating at VMPL0. Qemu exports the TPM space as a memory-mapped IO (MMIO)
region and traps all guest accesses to this region to forward the TPM commands
to the appropriate backend implementation - i.e., a passthrough device or an
emulator (swtpm). We cannot utilize the existing Qemu TPM interfaces for
communicating with the SVSM-vTPM as MMIO region of the TPM remains
unencrypted. A malicious hypervisor could alter the commands and perform a
man-in-the-middle (MITM) attack [36]. One could possibly implement a secure
communication channel using secure protocol and data model (SPDM) protocol to
prevent such attacks. However, it is much simpler to implement a new TPM
interface for Qemu.

We created a new Qemu TPM interface based on the command response buffer (CRB)
specification [23] in which the exported MMIO region is part of the guest
memory and is encrypted and accessible only to the confidential VM. However,
with this mechanism the accesses to this MMIO region are not trapped anymore
and the hypervisor is no longer aware on when to invoke the TPM backend
interface. Moreover, with SEV-SNP, the TPM region is marked as private to the
guest confidential VM in the reverse mapping table and a hypervisor cannot
perform a page remapping attack.

We modify the Linux CRB driver in the guest kernel to trigger an exit into the
hypervisor, after every write to the TPM MMIO region. Upon re-entry, the
hypervisor puts the vCPU in VMPL0 where SVSM-vTPM handler looks for TPM
command ready flag and inturn invokes the appropriate TPM command API to
formulate the response buffer. Then, the vCPU exits into VMPL1 and continues
with the execution of the guest VM. We also make modifications to the TPM
driver in OVMF to interact with our SVSM-vTPM.

###  4.3 vTPM state

A discrete physical TPM stores all the persistent state of the module inside
the chip’s non-volatile (NV) store which holds the seeds for generation of
endorsement key (EK), storage root key (SRK) and also retains other values
such as NV Index values, objects made persisted by the TPM user, and state
saved when TPM is shutdown. The TCG specification requires a TPM
implementation to have some amount of non-volatile storage for the operation
of the TPM [24].

As opposed to a physical TPM where the state of the TPM is securely stored
inside the TPM hardware chip inside a non-volatile RAM (NVRAM), a vTPM must
manage its state in software. Software vTPMs typically implement the NV store
in a disk-backed file [34, 32, 49, 57, 63, 74, 62]. Along with the software
that implements the vTPM, this NVRAM file is part of the trusted computing
base. When a vTPM is first initialized, the state file has to be created on-
the-fly or loaded from a file that is pre-created.

However, the state stored in the file needs to be secured against tampering
and rollback attacks [44]. This could be achieved by encrypting the NV store
file such that it could be decrypted only by the vTPM module. This design
calls for securely storing the secret key used to encrypt/decrypt the NV state
and inject it as a secret during the boot-up of vTPM module. This brings in
several complexities in the context of confidential computing as the secret
could only be injected during the launch phase. First the user has to verify
the pre-attestation report of the load-time components (i.e., firmware, OVMF,
etc.) before delivering the encrypted TPM state along with the key to decrypt
the TPM state. The booting of the platform is blocked, waiting for the user to
inject the secret. Additional care has to be taken to not give up the state to
a confidential VM that is under the control of an attacker.

#### Ephemeral vTPM

Instead, our design choice of using an ephemeral vTPM is much more simplistic
and pragmatic. The vTPM goes through the manufacturing process to generate a
fresh set of seeds, keys on every boot. We avoid all the problems of handling
persistent state, injecting it on every boot, and guarding the encrypted state
file by designing an ephemeral vTPM with no state. First, ephemeral vTPM is
simple to implement: the NV storage becomes a volatile storage and does not
preserve any values across power cycles. Second, it does not require any form
of secrets to boot-up the vTPM and the platform. Though there are downsides to
this design such as: secrets cannot be preserved across reboots, this offers
much more flexibility as there is no secret to guard against the
aforementioned attacks. Moreover, the programming environment for SVSM is
extremely constrained in terms of capabilities. To save the TPM state on
shutdown and to load the state on a reboot, the SVSM should implement
additional software to encrypt and decrypt the state file.

###  4.4 SVSM-vTPM provisioning

After launching the confidential VM, the hypervisor first loads and executes
the SVSM binary in VMPL0. Our modified SVSM follows the standard manufacturing
process of instantiating a vTPM instance as specified by the TPM2.0
specification [24]. First, we create a new endorsement key (EK) pair
$EK_{pub}$ and $EK_{priv}$ from random seeds. However, we do not create an
endorsement key certificate ($EK_{cert}$) or a platform certificate, as there
is no entity to sign these certificates.

A significant, and much under discussed problem in Confidential Computing is
seeding the random number generator. A VM when it boots has no natural sources
of entropy that are not under the control of the untrusted host. In an
ordinary VM, the x86 instructions RDRAND and RDSEED cause VMEXITs; however, in
confidential VMs, these instructions are guaranteed to provide direct access
to the CPU hardware random number in a way that the host cannot influence. We
use these instructions as the initial random number entropy source for
generating the random seeds.

###  4.5 Adding vTPM to the trust chain

Since our SVSM-vTPM module is instantiated with random seeds and does not come
with a manufacturer’s certificate to verify the identity of the TPM, we need
to ensure the following security properties:

  1. S1

Certify that the SVSM-vTPM is running in a real confidential VM on genuine AMD
hardware

  2. S2

Certify that the vTPM module is not tampered with.

  3. S3

Communicate $EK_{pub}$ in a secure, tamper-proof way.

To ensure these security properties, we rely on the attestation report from
the AMD-SP hardware.

#### SEV-SNP attestation report

Software running at any VMPL level can request an attestation report by
sending a message to the SEV firmware running inside the AMD-SP. The request
structure contains the VMPL level and 512-bits of space for user-provided data
which would be included as part of the attestation report signed by the AMD
hardware.

Figure 2 shows the steps involved in getting an attestation report. On
receiving a request to launch a VM, the platform loads the image and
cryptographically measures the contents of the image ( \raisebox{-0.9pt}{1}⃝
). Once the guest image is launched, the hypervisor puts the vCPU in VMPL0
mode passing control to the SVSM firmware (after \raisebox{-0.9pt}{2}⃝ ). The
SVSM firmware initializes the guest CPU, memory and sets up a pagetable for
execution and finally instantiates a vTPM. The vTPM is provisioned as
described in section 4.4. Then, the vTPM module requests an attestation report
by sending a SNP_REPORT_REQ message to the AMD-SP hardware (
\raisebox{-0.9pt}{3}⃝ ). We place the digest of the public part of the
generated endorsement key (i.e., $EK_{pub}$) in the user-data field of the
request to communicate the identity of the TPM to the guest VM. The request
message is encrypted with the appropriate VM platform communication key
(VMPCK) for that VMPL level and prepended with a message header which is
integrity protected with authenticated encryption (AEAD). The AMD-SP hardware
decrypts the message, verifies the integrity and responds with an attestation
report( \raisebox{-0.9pt}{4}⃝ ) that contains the _pre-attestation_
measurements, vmpl level and the user-data (i.e.,
$digest\left(EK_{pub}\right)$). We write this report into the NVIndex where
the TPM would normally place its EK certificate. We can retrieve the saved
attestation report at any point in time ( \raisebox{-0.9pt}{5}⃝ ) as long as
the guest VM is operational. If needed, the guest VM can also place a report
request to the AMD-SP hardware from other VMPL levels to generate a new
attestation report.

#### Ensuring S1

We can easily verify S1 because the attestation report is generated by the
AMD-SP processor and signed using AMD’s private keys. Verifying that the
attestation report is genuine implicitly guarantee that we obtained it from a
genuine AMD processor, within a confidential VM.

#### Ensuring S2

Before launching the confidential VM, the AMD-SP hardware measures all the
load-time binaries as part of the _pre-attestation_. This includes the SVSM
and our SVSM-vTPM code. By verifying these measurements that are included as
part of the attestation report, we can ensure that our SVSM-vTPM binary, and
anything else running in VMPL0, has not been tampered.

#### Ensuring S3

By verifying that the report request originated from VMPL0, we can confirm
that the report was requested by a legitimate SVSM-vTPM, based on S2. By
including the $digest(EK_{pub})$ as part of the attestation report (via user-
data field), we offer a tamper-proof way to communicate the identify of the
TPM ($EK_{pub}$) to the entities interacting with this specific vTPM. Since
$EK_{pub}$ and $EK_{priv}$ are generated from random seeds provided by the
hardware (i.e., RDRAND and RDSEED), as long as the generator is tamper-proof,
no entity can recreate $EK_{priv}$ and impersonate this vTPM.

##  5 Implementation

We base our implementation on the software stack recommended by AMD which is
publicly available on github [3]. It consists of qemu, open virtual machine
firmware (OVMF) and Linux kernel for both the host and the guest all of which
are modified to support the AMD SEV-SNP architecture and will eventually be
upstreamed. We make minor modifications to the open-source framework Keylime
[10] for performing remote attestation of VMs that has the SVSM-vTPM in the
root of trust.

To implement SVSM-vTPM, we extend the open-source SVSM implementation [12]
with a minimal C library (a stripped-down version of Musl [14]), WolfSSL
library [29] for cryptographic primitives, and Microsoft’s TPM that provides a
software reference implementation of TCG’s TPM 2.0 specification [15].

###  5.1 Software TCB

We add 1500 lines of code111measured using scc [22] to the existing SVSM
implementation in Rust. To implement vTPM, we utilize third party libraries: a
minimal C library (4100 LOC), WolfSSL crypto library (970 kLOC) and
Microsoft’s reference TPM implementation (32 kLOC) 222measured using David
Wheeler’s sloccount [15].

We trust the SVSM implementation and all the third-party crates that are part
of the dependency chain (i.e., recursive dependencies). We also assume that
WolfSSL and Microsoft’s TPM implementations are bug-free.

Figure 3: Remote attestation of a confidential VM using keylime and SVSM-vTPM

###  5.2 Remote attestation with Keylime

We use the Keylime package for remote attestation. Keylime is designed to
perform both boot-time and runtime attestation on a fleet of systems, using
the attested nodes’ TPM devices as the root of trust [10]. The Keylime
architecture is comprised of three major components: A Keylime _agent_ is
installed on every attested node. The agent announces itself with a Keylime
_registrar_ when it starts up. The Keylime _verifier_ is in charge of
performing attestations on every node.

#### Registration protocol

The purpose of Keylime registration is to record the availability of the
registering agent for attestation and to establish mutual trust between the
agent and the registrar. To this end the agent’s credentials are checked and
an attestation key is negotiated between the agent and the registrar for use
for subsequent attestation challenges. As shown in Figure 3, the agent
initiates the enrollment process by sending its TPM credentials - i.e., the
public part of its endorsement key (EK) and attestation identity key (AIK), as
well as the EK certificate, and the node’s UUID to the registrar. The
registrar verifies that the TPM’s identity and authenticity using the public
EK and the EK certificate. Next, the validity of the AIK is established
through the _MakeCredential/ActivateCredential_ function pair by using a
carefully constructed secret that can only survive the registrar to agent
roundtrip when both the TPM, AIK and UUID are authentic. Identity verification
of a normal TPM device involves checking that the EK certificate correctly
signs the public EK, and furthermore that the EK certificate ($EK_{cert}$) is
signed by a trusted root (such as a manufacturer key or an intermediary key).

#### Attestation protocol

Having successfully registered with the _registrar_ , the _agent_ is now ready
to service attestation challenges. The Keylime _verifier_ initiates the
attestation protocol by sending a TPM quote request to the _agent_ ,
containing a nonce (to guard against replay attacks) and a PCR mask (list of
PCRs). The _agent_ sends back the requested quote signed by the TPM, using the
AIK associated during the registration phase. In addition, a number of logs
(e.g. measured boot log, IMA log) are sent back with the quote. The _verifier_
validates the TPM quote by decrypting it with the registered AIK; validates
the logs by testing them against the PCRs contained in the quote; and finally
checks the contents of the logs against the attestation policy to render a
trustworthy/untrustworthy verdict.

#### Protocol changes to handle SVSM-vTPMs

Since Keylime is built around interaction with TPM devices, we needed to make
only minor modifications in the code to handle SVSM-vTPMs. Basically, we only
had to modify how the Keylime verifier checks the authenticity of a TPM device
(function check_ek). As mentioned above, a “normal” TPM device is
authenticated through its EK certificate, which signs the public EK and in
turn is verified by a manufacturer certificate. Keylime carries a list of
acceptable manufacturer certificates, and any TPM in use by Keylime has to be
signed by one of these. Our ephemeral SVSM-vTPM, by its very nature, is not
provisioned with an EK certificate. However, the (ephemeral) public EK is
signed by the SEV attestation report, which we validate by checking it against
the platform manufacturer’s signature (i.e., AMD). In order to minimize the
required changes in Keylime, we decided to simply replace the EK certificate
with an SEV attestation report ($Att_{report}$) in our SVSM-vTPM (that is, we
reuse the NVIndex in the TPM where the EK certificate normally resides). The
_agent_ reads and submits the attestation report instead of the EK certificate
during registration. The modified _registrar_ validates the attestation report
(ensuring that it is signed by an authentic AMD platform) instead of the
validating the EK certificate (marked by a different color in Figure 3). No
other parts of the registration/attestation protocols require changes for
correct Keylime function.

##  6 Evaluation

We ran all our experiments on publicly-available cloudlab infrastructure [64].
We utilize a Dell Poweredge R6525 server equipped with two AMD EPYC 7543
32-core processor and 256 GiB RAM. The host machine runs a 64-bit Ubuntu 20.04
Linux with v5.19 kernel and qemu v6.1.50, whereas the confidential guest VM
runs a 64-bit Ubuntu 22.04 Linux with a v5.17 kernel with open virtual machine
firmware (OVMF) version edk2-stable202208, all of which are modified to enable
SEV-SNP [3]. We have also evaluated our software stack on a Lenovo ThinkServer
equipped with an AMD EPYC 7763 64-core processor and 128 GiB RAM.

#### Performance overhead

To understand the overheads of commonly used TPM functionalities, we study the
performance of several TPM commands on SVSM-vTPM and compare that with a
vanilla virtual machine that utilizes a vTPM hosted by Qemu. We rely on
Qemu/KVM to launch both the regular and confidential VM. Qemu-vTPM setup uses
the native TPM CRB interface as its frontend with an swtpm backend where the
backed communicates with the vTPM running on the host userspace via a UNIX
socket interface. The SVSM-vTPM setup uses a modified TPM CRB interface as its
frontend and an SVSM backend (i.e., calling into VMPL0 to utilize the TPM
hosted by SVSM) running under VMPL0 inside the confidential VM environment.

Specifically, we compare the performance of four different TPM commands which
are essential for remote attestation, i.e., PCRREAD, PCREXTEND, TPM2_QUOTE,
CREATEPRIMARY. Also, we do not compare with a physical TPM as it would be
unfair since the physical TPMs are an order of magnitude slower than virtual
TPMs [63]. These TPM commands briefly do the following:

  * •

PCR read This command reads the platform configuration registers of the TPM. A
TPM may maintain multiple banks of PCR, where each bank is a collection of
PCRs extended with a specific hashing algorithm (e.g., sha1, sha256). In our
benchmark, we read all the PCR values from all the banks (i.e., sha1, sha256,
sha384).

  * •

PCR extend performs an extend operation on a specific PCR from a bank, i.e.,
it computes the hash of the old PCR value concatenated with the input data,
i.e., $PCR_{new}=hash(PCR_{old}||input\\_data)$. We extend a single PCR
register from a sha256 bank.

  * •

Quote A TPM quote contains a subset of PCRs from a bank and a nonce (to
prevent replay attacks) signed by the attestation key (AIK) of the TPM. We
request a quote of three PCRs (16-18) from two different banks (sha1 and
sha256).

  * •

Create primary The TPM command creates a primary object under the chosen
hierarchy (Endorsement, Platform, Owner or NULL) and loads it into the TPM.
The TPM returns only a context with which one can interact with this object
and the public and private portions of the key are not returned. We create an
ECC keypair with the default curve (ecc256).

We perform all the experiments by booting the confidential VM with the
corresponding setup (Qemu or SVSM), and invoke the TPM commands from the guest
user space using the tpm2-tools package [25]. For each TPM command, we ran the
benchmark for 3000 iterations. We ran these experiments three times to measure
the average latency ( Figure 4). SVSM-vTPM incurs 5x lower latency than Qemu-
vTPM on PCRREADs and to get a TPM2_QUOTE. We incur 1.8x and 3.5x lower latency
on PCREXTEND and CREATEPRIMARY TPM operations respectively. Both qemu-hosted
vTPM and SVSM-vTPM incur an exit into the hypervisor to communicate with the
TPM. However, our SVSM-vTPM suffers from much less overhead compared to the
qemu-hosted vTPM as the latter involves communicating with the TPM emulator
backend (i.e., swtpm) through the socket interface. On the other hand, the
physical TPMs are at least X times slower compared to the emulated ones as the
physical ones are often connected to the mainboard via a low-bandwidth bus
such as serial peripheral interface (SPI).

Figure 4: Performance overhead of SVSM-vTPM vs Qemu-vTPM

##  7 Security Analysis

The gist of our security argument is that we are tying an ephemeral vTPM to
the AMD-SP hardware’s root of trust to perform runtime attestation. In this
section we examine a number of potential security attacks and explain how our
design prevents them. Our hypothetical attacker’s goal would be to infiltrate
and alter a guest confidential VM without being detected by the remote
attestation system (Keylime).

#### Fake vTPM

The guest confidential VM boots with the SVSM firmware containing our SVSM-
vTPM as part of the VM launch process. The essence of this attack is that
after the system is booted and the keylime _agent_ is registered, an attacker
could spawn a new software vTPM in the guest userspace to hijack all the vTPM
commands and redirect to the newly spawned vTPM. The new fake software vTPM is
no longer running at a higher privilege level and can be controlled by the
attacker to forge TPM quotes in an attempt to authenticate fake boot and IMA
logs, and therefore hide unauthorized software alterations from keylime.

However, once the registration protocol is complete, the keylime registrar has
associated the $EK$ of our ephemeral vTPM with the $AIK$ that would be used
for signing the TPM quote. With the above redirection of TPM commands to a
fake vTPM an attacker would not be able to forge the TPM quote, as the fake
vTPM has no access to the private $AIK$ of the original vTPM, safely hidden by
VMPL0 in SVSM.

The attacker could possibly force the registration protocol to restart where
an attacker could feed the TPM credentials from the newly created vTPM. Again,
keylime would detect this because of the mismatch of the fake TPM’s $EK_{pub}$
with its digest in the attestation report. A fake attestation report cannot be
generated because the report contains the VMPL of the entity that requested
it, and the guest is not running at VMPL0.

#### Fake SEV-SNP attestation report

We save the attestation report requested by SVSM-vTPM at the same NVIndex as
the $EK_{cert}$ to make it available to the keylime agent. The essence of this
attack is that the attacker could overwrite this NVIndex with either garbage
data or another attestation report after compromising the guest. Garbage data
would be detected by the keylime _registrar_ , resulting in attestation
failure. When overwritten with a genuine attestation report, an attacker can
potentially change the identity of the vTPM, i.e., create another vTPM
(similar to Fake vTPM attack) with a new set of keys and record the new
$EK_{pub}$ as part of the user-data field of the attestation report. If
successful, they can perform all the attacks mentioned under the "Fake vTPM"
attack (i.e., spoof PCRs, forge quotes, etc).

Even though one could retrieve an attestation report from a different VM
privilege level, the platform guarantees that no one could spoof the VMPL
level in the attestation report as it could be generated only by the software
running inside VMPL0 (i.e., the keys for encrypting the request message is
available only at the corresponding level). Thus, the replaced attestation
report, if valid, would contain a VMPL level greater than 0. To prevent this
attack, we check the VMPL level while validating the attestation report to
ensure the requester VMPL level is set to zero.

An attacker can overwrite the attestation report NVIndex with a genuine
attestation report off another confidential VM or from a previous boot of this
confidential VM. Though the attestation report is signed by the AMD hardware,
the user-data will not match with the digest of $EK_{pub}$ we have inside the
SVSM-vTPM, making the attack detectable.

#### Confidential VMs with no SVSM

Though VMPL levels are supported in the SEV-SNP specification, it cannot be
enforced by the end-user on a provider-controlled environment. A malicious
cloud provider could host a regular SEV VM and pretend that it is running with
an SEV-SNP firmware. In this scenario, the confidential VM would run without
the SVSM firmware, where the entire guest operating system will run under
VMPL0. This makes it possible for a guest VM to generate its own attestation
report where the requester VMPL level is set to 0. To prevent this attack, we
need to verify that our confidential VM is booted with the SVSM firmware
running at VMPL0. The user can compute the measurement of their boot-time
binaries that includes the SVSM firmware running at VMPL0 and validate it
against the measurements reported in the attestation report provided by the
cloud provider. If the measurements do not match, the confidential VM is
likely booted without the SVSM firmware.

#### Weaknesses in random number generator

A weak HWRNG not only poses threat for the vTPM implementation, but also for
the software running inside the confidential VM. Failing to seed the random
number generator of a confidential VM correctly can result in cryptographic
key leakage[51], particularly in well documented random input signature
algorithms like ECDSA[50]. Furthermore, all vTPMs require a secure random
number generator to operate correctly because of their reliance on it for the
generation of ephemeral keys and nonces for secure functions. The problem is
particularly acute for an ephemeral vTPM because the TPM manufacturing stage
requires the generation of unguessable seeds which can only be achieved if
they are based on an entropy source which cannot be influenced in any way by
the host.

However, AMD hardware has suffered from a buggy HWRNG in the past where RDRAND
instruction was always giving out a constant value instead of a random number
[7]. An attacker could exploit a weak or buggy HWRNG implementation to guess
the initial seeds of the vTPM and create the same secret keys as the vTPM. For
example, by guessing the attestation key, one could forge TPM quotes and break
the guarantees of remote attestation. To be resilient to such hardware bugs,
we can seed the random number generator with additional sources of entropy
such as the hash of a key derived by the AMD-SP upon user’s request along with
the RDSEED instruction.

##  8 Discussion

#### Full disk encryption

Full disk encryption (FDE) protects the confidentiality and integrity of data
at-rest. To prevent accidental disclosure of the secret key (e.g., disk
encryption key), it is a standard practice to encrypt the secret key (_wrap_
operation) such that it can be decrypted only by the TPM (_unwrapping_). The
wrapping key (i.e., the key which wraps the secret) is often the storage root
key (SRK) present in the TPM.

However, in our ephemeral vTPM, there are no persistent storage keys in the
TPM to support unwrapping of keys. To support FDE on such systems, we create
an intermediary storage key $K_{iSK}$ ( a restricted decryption key with
sensitiveDataOrigin [45]). Now, we perform a TPM _seal_ operation on the disk
encryption key by parenting it to the storage key ($K_{iSK}$) we just created,
outputting a sealed blob which can be unsealed only by a TPM with the same
key. On platform boot, the vTPM would generate an ephemeral endorsement key
and an ephemeral storage root key (eSRK). By retrieving the public part of the
eSRK, we can wrap the intermediary key $K_{iSK}$ with $eSRK_{pub}$ to create a
wrapped key that can be decrypted only by our vTPM. It has to be noted that
all the above operations can be performed on any TPM, i.e., the user need not
necessarily perform these on the vTPM of the confidential VM. Now, the disk
encryption key is wrapped to the parent key and the parent is in turn wrapped
to the eSRK, forming a hierarchy under the ephemeral storage root key. It is
also possible to wrap the parent key with $EK_{pub}$ instead to create a
hierarchy under the ephemeral endorsement key.

As both the disk encryption key and its parent key are wrapped for our
specific vTPM, they are no longer a secret and can be delivered to the
confidential VM in the clear. Since the sealed disk encryption key is
invariant, we can embed this into the initrd. Finally, we can deliver the
wrapped parent key to the confidential VM once we have performed the initial
attestation of the platform to ensure its trustworthiness.

#### Storing secrets

We cannot store secrets directly by wrapping the keys on our ephemeral SVSM-
vTPM as the EK and SRK would be newly generated on every boot. One could use a
similar technique we used for FDE to form a hierarchy of keys under an
intermediary storage key. Once the system is booted, we can parent the
intermediary key to the ephemeral SRK or EK forming a hierarchy under the
chosen key. Using this technique, one could store a hierarchy of keys, as we
do on a regular persistent TPM.

##  9 Related work

#### Cloud vTPMs

Cloud providers offering confidential VMs typically provide virtual TPM device
that would serve as a root-of-trust and could also be used for remote
attestation. Google cloud only offers plain SEV confidential VMs and offers
measured boot attestation via a vTPM managed by the hypervisor [27]. Microsoft
Azure cloud relies on azure attestation service for attesting confidential VMs
[13] that generates a token to decrypt the vTPM state and the disk, hinting
that Microsoft may have their custom firmware based on SVSM specification
(i.e., inside VMPL0) with a persistent vTPM for attesting SEV-SNP VMs. Alibaba
cloud offers vTPM support on their elastic compute service VMs [1]. Amazon AWS
provides Nitro TPM, a virtual TPM implementation conforming to the TPM 2.0
specification as part of their EC2 offering [6]. Some of these providers use a
qemu-backed vTPM that runs on the host, which requires trusting the cloud
provider. Also, there is very limited public knowledge on how these cloud
vTPMs are designed and the security guarantees of it. In contrast, we plan to
publish the source code of SVSM-vTPM implementation that is built on top of
other standard opensource components (i.e., Qemu, Linux, and Keylime). As our
SVSM-vTPM rely only on the hardware-protected isolation environment offered by
the AMD-SP hardware, by bringing their own SVSM firmware, a user can
completely eliminate the need for trusting the cloud provider.

#### TEE-based vTPMs

CoCoTPM proposes a unified architecture for attestation of confidential VMs
where the hypervisor launches a confidential VM that acts as a vTPM manager
and handles all the vTPM instances [62]. They require TLS for securing the
communication channel between a confidential VM and its vTPM. Though the vTPM
is running under a TEE, a central vTPM manager suffers from several attacks
ranging from denial of service to colluding with other confidential VMs, on
the other hand launching a dedicated CoCoTPM for every confidential VM results
in wastage of architectural resources as the number of address space
identifiers (ASIDs) are limited.

Several projects rely on running vTPM under isolation provided by other
hardware TEE mechanisms such as Intel SGX [74, 69, 75] and ARM Trustzone [63].
SvTPM aims to protect against NVRAM replacement, and rollback attacks [74] by
running the vTPM inside an SGX enclave for KVM-based VMs, whereas eTPM manages
several enclave vTPMs in a Xen environment and relies on a physical TPM to
provide root-of-trust [69], similar to Berger et al. [34]. In contrast, our
SVSM-vTPM architecture equips each confidential VM with their own private vTPM
instance by leveraging the SVSM architecture that implements VM privilege
levels. Also, by implementing an ephemeral vTPM, we completely eliminate the
classes of attacks that come with state protection.

#### Trusted execution environments

Arm introduced confidential compute architecture (CCA) with their Armv9-A
architecture where the processor provides an isolated hardware execution
environment called _Realms_ , for hosting entire VMs in a secure space [9].
Similar to other TEEs [2, 8] they offer pre-attestation of realms and can do
measured boot with their hardware enforced security (HES) module specification
[5] which serves as the root-of-trust [18, 33].

Intel, with their trust domain extensions (TDX) introduced their own version
of hardware-isolated encrypted virtual machines called trusted domains (TDs).
Intel TDX relies on an SGX-based quoting enclave called the TD-quoting enclave
to perform remote attestation of trusted domains [8]. However, SGX suffered
from numerous vulnerabilities in the past [59] where researchers were able to
extract the SGX quoting enclave’s attestation keys through micro-architectural
side-channel attacks to forge attestation reports [72]. The attestation keys
used by these quoting enclave are long-lived, and when leaked, affect millions
of devices. In our design, we do not have any secrets to guard as the
attestation keys are ephemeral.

##  10 Conclusions

The landscape of cloud security is changing with the growing desire to remove
the cloud provider from the trust domain. Hardware vendors lay foundation for
implementing this vision through a collection of mechanisms that ensure
confidentiality of a cloud execution, i.e., encryption of VM memory, but,
unfortunately, lack support for ensuring runtime integrity. Our work develops
a novel approach for virtualizing the hardware root-of-trust through a
combination of hardware mechanisms and a new ephemeral approach to the
management of the TPM state. We demonstrate how an ephemeral vTPM can be used
for providing remote attestation of a confidential VM. In the spirit of
transparency, our implementation is in the open source, and can be audited,
verified, and extended. As more and more cloud providers are gearing up for
offering confidential VMs, we believe our SVSM-vTPM architecture would provide
a reference point for implementing a vTPM on SEV-SNP infrastructure. While our
implementation is tied to SEV-SNP, there is no real reason why it could not be
replicated in other TEEs. The only requirement is to deploy ephemeral vTPMs
inside a secure enclave that is isolated from both the rest of the hardware
(i.e. encrypted) and from the guest (the equivalent of what VMPLs provide).

## References

  * [1] Alibaba Cloud Security White Paper.  https://alicloud-common.oss-ap-southeast-1.aliyuncs.com/2021/HTK%20Region/whitepaper/Alibaba%20Cloud%20Security%20Whitepaper.pdf.  Online; accessed Dec 17, 2022.
  * [2] AMD Memory encryption.  https://amd.wpenginepowered.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v9-Public.pdf.  Online; accessed Dec 19, 2022.
  * [3] AMD Secure Encrypted Virtualization.  https://github.com/AMDESE.  Online; accessed Nov 11, 2022.
  * [4] AMD SEV-SNP: Strengthening VM Isolation with Integrity Protection and More.  https://www.amd.com/system/files/TechDocs/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf.  Online; accessed Dec 17, 2022.
  * [5] Arm CCA Security Model 1.0.  https://developer.arm.com/documentation/DEN0096/latest.  Online; accessed Dec 17, 2022.
  * [6] AWS Nitro System.  https://aws.amazon.com/ec2/nitro/.  Online; accessed Dec 17, 2022.
  * [7] How a months-old AMD microcode bug destroyed my weekend.  https://arstechnica.com/gadgets/2019/10/how-a-months-old-amd-microcode-bug-destroyed-my-weekend/.  Online; accessed Dec 7, 2022.
  * [8] Intel Trust Domain CPU Architectural Extensions.  https://www.intel.com/content/dam/develop/external/us/en/documents/intel-tdx-cpu-architectural-specification.pdf.  Online; accessed Dec 17, 2022.
  * [9] Introducing Arm Confidential Compute Architecture.  https://developer.arm.com/documentation/den0125/0200.  Online; accessed Dec 19, 2022.
  * [10] Keylime.  https://github.com/keylime/keylime.  Online; accessed Nov 11, 2022.
  * [11] KVM CVEs.  https://nvd.nist.gov/vuln/search.  Online; accessed Dec 17, 2022.
  * [12] Linux SVSM for secure x86 virtualization in Rust.  https://github.com/AMDESE/linux-svsm.  Online; accessed Nov 11, 2022.
  * [13] Microsoft Azure Attestation.  https://learn.microsoft.com/en-us/azure/attestation/overview.  Online; accessed Dec 6, 2022.
  * [14] musl libc.  https://musl.libc.org/.  Online; accessed Dec 17, 2022.
  * [15] Official TPM 2.0 Reference Implementation (by Microsoft).  https://github.com/microsoft/ms-tpm-20-ref.  Online; accessed Dec 17, 2022.
  * [16] OpenSSL.  https://github.com/openssl/openssl.  Online; accessed Nov 11, 2022.
  * [17] Protecting VM register state with SEV-ES.  https://www.amd.com/system/files/TechDocs/Protecting%20VM%20Register%20State%20with%20SEV-ES.pdf.  Online; accessed Dec 17, 2022.
  * [18] Runtime Security Subsystem (RSS).  https://tf-m-user-guide.trustedfirmware.org/platform/arm/rss/readme.html.  Online; accessed Dec 17, 2022.
  * [19] Secure VM Service Module for SEV-SNP Guests.  https://developer.amd.com/wp-content/resources/58019.pdf.  Online; accessed Dec 17, 2022.
  * [20] Security Protocol and Data Model (SPDM) Specification.  https://www.dmtf.org/sites/default/files/standards/documents/DSP0274_1.1.0.pdf.  Online; accessed Dec 17, 2022.
  * [21] SEV-ES Guest-Hypervisor Communication Block Standardization.  https://developer.amd.com/wp-content/resources/56421.pdf.  Online; accessed Dec 17, 2022.
  * [22] Sloc Cloc and Code (scc).  https://github.com/boyter/scc.  Online; accessed Dec 17, 2022.
  * [23] TCG PC Client Platform TPM Profile Specification for TPM 2.0. v1.05r14.  https://trustedcomputinggroup.org/wp-content/uploads/PC-Client-Specific-Platform-TPM-Profile-for-TPM-2p0-v1p05p_r14_pub.pdf.  Online; accessed Dec 17, 2022.
  * [24] TPM 2.0 library.  https://trustedcomputinggroup.org/resource/tpm-library-specification/.  Online; accessed Dec 17, 2022.
  * [25] Trusted Platform Module (TPM2.0) tools.  https://github.com/tpm2-software/tpm2-tools.  Online; accessed Dec 17, 2022.
  * [26] Validating instances using Cloud Monitoring.  https://cloud.google.com/compute/confidential-vm/docs/monitoring.  Online; accessed Dec 6, 2022.
  * [27] Virtual Trusted Platform Module for Shielded VMs: security in plaintext.  https://cloud.google.com/blog/products/identity-security/virtual-trusted-platform-module-for-shielded-vms-security-in-plaintext.  Online; accessed Dec 6, 2022.
  * [28] VMWare CVEs.  https://www.vmware.com/security/advisories.html.  Online; accessed Dec 17, 2022.
  * [29] wolfSSL Embedded SSL/TLS Library.  https://github.com/wolfSSL/wolfssl/.  Online; accessed Dec 17, 2022.
  * [30] Xen vulnerability statistics.  https://www.cvedetails.com/product/23463/XEN-XEN.html?vendor_id=6276.  Online; accessed Dec 17, 2022.
  * [31] AMD.  Secure Encrypted Virtualization API Version 0.24.  https://www.amd.com/system/files/TechDocs/55766_SEV-KM_API_Specification.pdf.  Online; accessed Dec 17, 2022.
  * [32] Melvin J Anderson, Micha Moffie, Chris I Dalton, et al.  Towards Trustworthy Virtualization Environments: Xen Library OS Security Service Infrastructure.  Trusted Systems Laboratory, HP Laboratories Bristol, pages 88–111, 2007.
  * [33] Tamas Ban.  Attestation and Measured Boot.  https://www.trustedfirmware.org/docs/Attestation_and_Measured_Boot.pdf.  Online; accessed Dec 17, 2022.
  * [34] Stefan Berger, Ramón Cáceres, Kenneth A. Goldman, Ronald Perez, Reiner Sailer, and Leendert van Doorn.  vTPM: Virtualizing the Trusted Platform Module.  In Proceedings of the 15th Conference on USENIX Security Symposium - Volume 15, USENIX-SS’06. USENIX Association, 2006.
  * [35] Felix Bohling, Tobias Mueller, Michael Eckel, and Jens Lindemann.  Subverting Linux’ Integrity Measurement Architecture.  In Proceedings of the 15th International Conference on Availability, Reliability and Security, ARES ’20. Association for Computing Machinery, 2020.
  * [36] Jeremy Boone.  TPM Genie: Interposer Attacks Against the Trusted Platform Module Serial Bus.  White paper, NCC Group, March 2018.  https://www.nccgroup.com/globalassets/about-us/us/documents/tpm-genie.pdf.
  * [37] Haogang Chen, Yandong Mao, Xi Wang, Dong Zhou, Nickolai Zeldovich, and M. Frans Kaashoek.  Linux kernel vulnerabilities: state-of-the-art defenses and open problems.  In APSys, pages 5:1–5:5, 2011.
  * [38] George Coker, Joshua Guttman, Peter Loscocco, Amy Herzog, Jonathan Millen, Brian O’Hanlon, John Ramsdell, Ariel Segall, Justin Sheehy, and Brian Sniffen.  Principles of remote attestation.  International Journal of Information Security, 10(2):63–81, 2011\.
  * [39] Jonathan Corbet.  Supervisor mode access prevention.  https://lwn.net/Articles/517475/, 2012.  Online; accessed Dec 17, 2022.
  * [40] Intel Corporporation.  Intel® 64 and IA-32 Architectures Software Developer’s Manual, 2022.  https://software.intel.com/content/www/us/en/develop/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4.html.
  * [41] Victor Costan and Srinivas Devadas.  Intel SGX Explained.  Cryptology ePrint Archive, Paper 2016/086, 2016.  https://eprint.iacr.org/2016/086.
  * [42] Crispin Cowan, Calton Pu, Dave Maier, Heather Hinton, and Jonathan Walpole.  StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks.  In USENIX Security Symposium, 1998.
  * [43] Stephen Fischer.  Supervisor Mode Execution Protection.  NSA Trusted Computing Conference, 2011.
  * [44] Trusted Computing Group.  Trusted Platform Module Library Part 1: Architecture.  [46], chapter 37.7.2 "External NV", pages 232–233.
  * [45] Trusted Computing Group.  Trusted Platform Module Library Part 1: Architecture.  [46], chapter 25.2.3 "sensitiveDataOrigin", page 198\.
  * [46] Trusted Computing Group.  Trusted Platform Module Library Part 1: Architecture, level 00 revision 01.59 edition, Nov 2019.
  * [47] Vivek Haldar, Deepak Chandra, and Michael Franz.  Semantic Remote Attestation: A Virtual Machine Directed Approach to Trusted Computing.  In Proceedings of the 3rd Conference on Virtual Machine Research And Technology Symposium - Volume 3, VM’04, page 3. USENIX Association, 2004\.
  * [48] Guerney D. H. Hunt, Ramachandra Pai, Michael V. Le, Hani Jamjoom, Sukadev Bhattiprolu, Rick Boivie, Laurent Dufour, Brad Frey, Mohit Kapur, Kenneth A. Goldman, Ryan Grimm, Janani Janakirman, John M. Ludden, Paul Mackerras, Cathy May, Elaine R. Palmer, Bharata Bhasker Rao, Lawrence Roy, William A. Starke, Jeff Stuecheli, Enriquillo Valdez, and Wendel Voigt.  Confidential Computing for OpenPOWER.  In Proceedings of the Sixteenth European Conference on Computer Systems, EuroSys ’21, page 294–310. Association for Computing Machinery, 2021\.
  * [49] Xin Jin, Li-na Wang, Rong-wei Yu, Peng Kou, and Cheng-lin Shen.  Administrative Domain: Security Enhancement for Virtual TPM.  In 2010 International Conference on Multimedia Information Networking and Security, pages 767–771, 2010.
  * [50] Don Johnson, Alfred Menezes, and Scott Vanstone.  The elliptic curve digital signature algorithm (ECDSA).  International journal of information security, 1(1):36–63, 2001\.
  * [51] John Kelsey, Bruce Schneier, David Wagner, and Chris Hall.  Cryptanalytic attacks on pseudorandom number generators.  In International workshop on fast software encryption, pages 168–188. Springer, 1998.
  * [52] Volodymyr Kuznetsov, László Szekeres, Mathias Payer, George Candea, R. Sekar, and Dawn Song.  Code-Pointer Integrity.  In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, OSDI’14, page 147–163, USA, 2014. USENIX Association.
  * [53] Andrew Kwong, Daniel Genkin, Daniel Gruss, and Yuval Yarom.  RAMBleed: Reading Bits in Memory Without Accessing Them.  In 2020 IEEE Symposium on Security and Privacy (SP), pages 695–711, 2020.
  * [54] Mengyuan Li, Luca Wilke, Jan Wichelmann, Thomas Eisenbarth, Radu Teodorescu, and Yinqian Zhang.  A Systematic Look at Ciphertext Side Channels on AMD SEV-SNP.  In 2022 IEEE Symposium on Security and Privacy (SP), pages 337–351, 2022.
  * [55] Akash Malhotra.  AMD RYZEN pro 5000 series mobile processors.  https://www.amd.com/system/files/documents/amd-security-white-paper.pdf.  Online; accessed Dec 17, 2022.
  * [56] Mengyuan Li and Yinqian Zhang and Huibo Wang and Kang Li and Yueqiang Cheng.  CIPHERLEAKS: Breaking Constant-time Cryptography on AMD SEV via the Ciphertext Side Channel.  In 30th USENIX Security Symposium (USENIX Security 21), pages 717–732. USENIX Association, August 2021.
  * [57] Derek Gordon Murray, Grzegorz Milos, and Steven Hand.  Improving Xen security through disaggregation.  In Proceedings of the fourth ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, pages 151–160, 2008.
  * [58] Eugene D Myers.  Using the Intel STM for Protected Execution.  https://www.platformsecuritysummit.com/2018/speaker/myers/STMPE2Intelv84a.pdf, 2018\.  Online; accessed Dec 17, 2022.
  * [59] Alexander Nilsson, Pegah Nikbakht Bideh, and Joakim Brorsson.  A survey of published attacks on Intel SGX.  arXiv preprint arXiv:2006.13598, 2020.
  * [60] Wojciech Ozga, Do Le Quoc, and Christof Fetzer.  TRIGLAV: Remote Attestation of the Virtual Machine’s Runtime Integrity in Public Clouds.  In 2021 IEEE 14th International Conference on Cloud Computing (CLOUD), pages 1–12, 2021.
  * [61] Bryan Parno, Jonathan M. McCune, and Adrian Perrig.  Bootstrapping Trust in Commodity Computers.  In 2010 IEEE Symposium on Security and Privacy, pages 414–429, 2010\.
  * [62] Joana Pecholt and Sascha Wessel.  CoCoTPM: Trusted Platform Modules for Virtual Machines in Confidential Computing Environments.  In Proceedings of the 38th Annual Computer Security Applications Conference, ACSAC ’22, page 989–998. Association for Computing Machinery, 2022\.
  * [63] Himanshu Raj, Stefan Saroiu, Alec Wolman, Ronald Aigner, Jeremiah Cox, Paul England, Chris Fenner, Kinshuman Kinshumann, Jork Loeser, Dennis Mattoon, Magnus Nystrom, David Robinson, Rob Spiger, Stefan Thom, and David Wooten.  fTPM: A Software-Only Implementation of a TPM Chip.  In Proceedings of the 25th USENIX Conference on Security Symposium, SEC’16, page 841–856. USENIX Association, 2016.
  * [64] Robert Ricci, Eric Eide, and CloudLab Team.  Introducing CloudLab: Scientific Infrastructure for Advancing Cloud Architectures and Applications.  ; login:: the magazine of USENIX & SAGE, 39(6):36–38, 2014.
  * [65] Reiner Sailer, Xiaolan Zhang, Trent Jaeger, and Leendert Van Doorn.  Design and implementation of a TCG-based integrity measurement architecture.  In USENIX Security symposium, volume 13, pages 223–238, 2004.
  * [66] Vincent Scarlata, Carlos Rozas, Monty Wiseman, David Grawrock, and Claire Vishik.  TPM Virtualization: Building a General Framework, pages 43–56.  Vieweg+Teubner, 2008.
  * [67] Hovav Shacham, Matthew Page, Ben Pfaff, Eu-Jin Goh, Nagendra Modadugu, and Dan Boneh.  On the effectiveness of address-space randomization.  In Proceedings of the 11th ACM conference on Computer and Communications Security, pages 298–307, 2004.
  * [68] Frederic Stumpf and Claudia Eckert.  Enhancing Trusted Platform Modules with Hardware-Based Virtualization Techniques.  In 2008 Second International Conference on Emerging Security Information, Systems and Technologies, pages 1–9, 2008.
  * [69] Haonan Sun, Rongyu He, Yong Zhang, Ruiyun Wang, Wai Hung Ip, and Kai Leung Yung.  eTPM: A Trusted Cloud Platform Enclave TPM Scheme Based on Intel SGX Technology.  Sensors, 18(11), 2018.
  * [70] Chihiro Tomita, Makoto Takita, Kazuhide Fukushima, Yuto Nakano, Yoshiaki Shiraishi, and Masakatu Morii.  Extracting the Secrets of OpenSSL with RAMBleed.  Sensors, 22(9), 2022.
  * [71] Chia-Che Tsai, Donald E. Porter, and Mona Vij.  Graphene-SGX: A Practical Library OS for Unmodified Applications on SGX.  In Proceedings of the 2017 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’17, page 645–658. USENIX Association, 2017\.
  * [72] Jo Van Bulck, Marina Minkin, Ofir Weisse, Daniel Genkin, Baris Kasikci, Frank Piessens, Mark Silberstein, Thomas F Wenisch, Yuval Yarom, and Raoul Strackx.  Foreshadow: Extracting the Keys to the Intel SGX Kingdom with Transient Out-of-Order Execution.  In 27th USENIX Security Symposium (USENIX Security 18), pages 991–1008, 2018.
  * [73] Arjan van de Ven.  New Security Enhancements in Red Hat Enterprise Linux v.3, update 3\.  https://static.redhat.com/legacy/f/pdf/rhel/WHP0006US_Execshield.pdf.
  * [74] Juan Wang, Chengyang Fan, Jie Wang, Yueqiang Cheng, Yinqian Zhang, Wenhui Zhang, Peng Liu, and Hongxin Hu.  SvTPM: A Secure and Efficient vTPM in the Cloud.  arXiv:1905.08493, 2019.
  * [75] Juan Wang, Feng Xiao, Jianwei Huang, Daochen Zha, Chengyang Fan, Wei Hu, and Huanguo Zhang.  A Security-Enhanced vTPM 2.0 for Cloud Computing.  In Information and Communications Security, pages 557–569. Springer International Publishing, 2018.
  * [76] Stephen Weis.  Protecting data in-use from firmware and physical attacks.  Black Hat, 2014.
  * [77] Richard Wilkins and Brian Richardson.  UEFI secure boot in modern computer security solutions.  In UEFI forum, pages 1–10, 2013.