Your Windows Kernel Can Be Compromised. What VBS Enclaves Do Next
Your Windows Kernel Can Be Compromised. What VBS Enclaves Do Next.
For decades, operating-system security was built around a relatively simple assumption:
the operating system is trusted.
Applications run above the kernel.
The kernel controls memory, processes, devices and access to resources.
If an application is compromised, the attacker is supposed to hit another boundary before reaching the kernel.
If the attacker reaches the kernel, the machine is considered deeply compromised.
Modern Windows is changing that model.
Virtualization-Based Security already assumes something uncomfortable: the Windows kernel itself may become compromised.
Instead of asking the kernel to protect every security-sensitive component, Windows can use the hypervisor to create an isolated environment outside the normal Windows trust domain.
And now Microsoft is taking that concept further with VBS Enclaves.
A VBS enclave is a software-based trusted execution environment inside a host application’s address space. Its sensitive code and data are isolated from the host application and the rest of the system. Microsoft currently supports VBS Enclaves on Windows 11 build 26100.2314 or later and Windows Server 2025 or later.
This is not another antivirus feature.
It is a change in the architecture of trust.
Traditional model
Application
|
v
Windows
Kernel
|
v
Hardware
VBS changes it:
Application
|
v
Windows Kernel
|
X
|
Hypervisor
|
v
VBS Security Environment
And VBS Enclaves push application secrets even further:
Host Application
|
+----------------------+
| |
v v
Normal Memory VBS Enclave
| |
| Sensitive Code
| Sensitive Data
| |
+----------X-----------+
|
Isolation
The important question is no longer:
How do we prevent every compromise?
It becomes:
What must remain protected after the compromise has already happened?
The old security boundary is no longer enough
Consider a normal Windows application.
It has:
Process
├── Code
├── Heap
├── Stack
├── DLLs
├── Credentials
├── Encryption keys
└── Application secrets
If an attacker gets arbitrary code execution inside that process, they potentially inherit the process’s access to memory.
That is the fundamental problem.
A browser exploit doesn’t necessarily need to become SYSTEM immediately.
A malicious plugin doesn’t need kernel access.
A compromised enterprise application doesn’t need administrator privileges to steal secrets belonging to the application itself.
If the secret is stored inside the same trust domain as the compromised application, the attacker may simply read it.
The traditional architecture looks like:
+--------------------------------------+
| Host Application |
| |
| Code Data Secrets |
| | | | |
| +---------+----------+ |
| TRUST |
+--------------------------------------+
Compromise the process and the attacker enters the trust domain.
VBS Enclaves change that relationship.
+--------------------------------------+
| Host Application |
| |
| Normal Code Normal Data |
| |
| X |
| | |
| +-------------+ |
| | VBS Enclave | |
| | | |
| | Secret Key | |
| | Sensitive | |
| | Operations | |
| +-------------+ |
+--------------------------------------+
The application becomes a host rather than the ultimate security boundary.
That is a much more interesting architecture.
VBS already assumes the kernel can be hostile
The important part of VBS is not simply that it uses virtualization.
It changes the trust model.
Microsoft describes VBS as using hardware virtualization and the Windows hypervisor to create an isolated virtual environment that becomes a root of trust for the operating system. The design explicitly assumes that the Windows kernel can potentially be compromised.
That statement matters.
The traditional model is:
Hardware
|
Kernel
|
Applications
The security hierarchy is essentially built around the kernel.
VBS introduces another layer:
Hardware
|
Hypervisor
|
+--------------------------+
| Secure VBS environment |
+--------------------------+
|
Windows kernel
|
Applications
The kernel is no longer the absolute bottom of the security model.
The hypervisor sits underneath it.
This is one of the reasons VBS can protect security mechanisms from certain classes of kernel compromise.
Memory Integrity, also known as HVCI, runs kernel-mode code integrity inside the isolated VBS environment. Microsoft describes this as a mechanism for preventing untrusted or unsigned kernel code from being loaded and for restricting memory states that could otherwise be abused by kernel-level malware.
The architecture therefore becomes:
Application
|
v
Windows
|
v
Kernel
|
X
|
Hypervisor
|
v
Protected Security Environment
But there is still a problem.
What happens when an application contains its own crown jewels?
A compromised application can still be dangerous
Imagine an enterprise application that handles:
API credentials
Private keys
Authentication tokens
Encryption keys
Payment secrets
Database credentials
Signing keys
Customer secrets
The application might run perfectly with:
No administrator privileges
No SYSTEM privileges
No kernel access
Yet an exploit inside the application could still expose all of those secrets.
This is because:
Application compromise
|
v
Application memory
|
v
Application secrets
The attacker doesn’t need to escalate.
The secret is already there.
This is one of the fundamental limitations of conventional process isolation.
A process boundary protects one process from another.
It does not automatically protect the application’s secrets from the application itself after the process has been compromised.
VBS Enclaves are designed for exactly this kind of problem.
An enclave creates a smaller trust domain
The key idea is simple.
Do not put the secret inside the normal process.
Put the sensitive operation inside the enclave.
Instead of:
Application
|
+-- read key
|
+-- decrypt data
|
+-- return plaintext
you build:
Application
|
| request
v
+------------------+
| VBS Enclave |
| |
| Private key |
| Crypto operation |
+------------------+
|
| result
v
Application
The application asks the enclave to perform a sensitive operation.
The key does not need to be exposed to the normal application memory.
This is a classic trusted execution environment design.
Microsoft describes an enclave as an isolated region of code and data within an application’s address space, where code outside the enclave cannot directly access the enclave’s protected data.
The critical design principle is:
Move the secret, not just the permission.
Giving an application access to a secret and then trying to restrict what the application does with it is often weaker than designing the application so it never receives the raw secret in the first place.
The host becomes an untrusted coordinator
This changes application architecture.
Traditional application:
Application
|
+--------------+--------------+
| | |
v v v
Secrets Business Network
Logic
Enclave-based architecture:
Host
|
+--------+--------+
| |
v v
Business Logic Network
|
|
v
+-------------+
| VBS Enclave |
| |
| Keys |
| Sensitive |
| Operations |
+-------------+
The host becomes something closer to an untrusted orchestrator.
It can request an operation.
It can receive an answer.
But it should not automatically receive the secret material that made the operation possible.
This is a very different threat model.
Why this matters after RCE
Let’s take a realistic attack chain.
Internet
|
v
Application
|
v
Remote Code Execution
|
v
Attacker controls host process
Traditional architecture:
Compromised Process
|
+---- memory
|
+---- credentials
|
+---- private keys
|
+---- tokens
Enclave architecture:
Compromised Process
|
+---- normal memory
|
+---- application state
|
X
|
+---- enclave secrets
|
v
Protected Environment
The attacker may still own the host.
That is important.
An enclave is not designed to magically restore the integrity of the compromised application.
Instead, it tries to prevent the compromise from automatically exposing the enclave’s protected state.
That distinction is critical.
VBS Enclaves are not virtual machines
A common misunderstanding would be to describe an enclave as:
a tiny virtual machine.
It isn’t.
A VM contains an operating system environment:
VM
|
+-- Guest Kernel
+-- Drivers
+-- Services
+-- Applications
An enclave is much smaller:
Host Process
|
+-- VBS Enclave
|
+-- Protected Code
+-- Protected Data
Microsoft explicitly describes VBS Enclaves as software-based TEEs that use the Windows hypervisor and do not require special enclave hardware.
The objective is not to create another operating system.
The objective is to create a much smaller trusted computing base.
And smaller trust domains are easier to reason about.
The hypervisor becomes part of the security boundary
Once VBS is involved, the hypervisor becomes critical.
The Windows kernel can no longer be considered the lowest security layer.
The model becomes:
Application
|
Windows User Mode
|
Windows Kernel
|
=====================
Hypervisor
=====================
|
Protected VBS World
|
Hardware
Microsoft documents this explicitly when describing VBS resource protection: the hypervisor must assume that kernel-mode code could be malicious and must therefore protect hardware resources that could undermine VBS security.
This is a profound architectural shift.
The hypervisor has to police the kernel.
For example, VBS can control access to processor MSRs that could otherwise be abused by malicious kernel-mode code. Microsoft documents hypervisor enforcement and event logging for blocked restricted MSR access.
So the trust hierarchy becomes:
Application
↓
User Mode
↓
Kernel
↓
Hypervisor
↓
Hardware
But only the lower layer is supposed to have the authority to protect the upper security boundary.
This is defense against a compromised kernel, not just malware
That distinction is worth emphasizing.
Traditional endpoint security often thinks in terms of:
Malware
|
v
Detection
|
v
Blocking
VBS takes another approach:
Assume compromise
|
v
Protect the security boundary
|
v
Limit what the compromised layer can modify
This is much closer to modern hardware-backed security architecture.
It assumes attackers will eventually find bugs.
The goal is to prevent one bug from collapsing the entire trust model.
VBS Enclaves add a second application-level boundary
VBS already protects parts of the operating system.
VBS Enclaves extend the concept into application design.
Think about the difference:
VBS
Protect Windows security
from a compromised Windows kernel.
versus:
VBS Enclave
Protect sensitive application code/data
from a compromised host application.
Together:
Application
|
+--------+--------+
| |
v v
Normal Host VBS Enclave
| |
v v
Windows Kernel Sensitive Data
| |
+--------X--------+
|
Hypervisor
|
Hardware
The result is a hierarchy of trust domains.
The enclave must be signed
Security boundaries are useless if attackers can replace the code running inside them.
This is why VBS enclave deployment includes code signing.
Microsoft’s VBS Enclave development documentation requires enclave DLLs to be signed before successful loading. The signature carries information used to derive the enclave’s Author ID.
The build pipeline therefore includes:
Source
|
v
Compile
|
v
Enclave DLL
|
v
Integrity configuration
|
v
Signing
|
v
Load
This makes the supply chain part of the enclave security model.
A compromised build pipeline is therefore not a minor DevOps problem.
It can become a direct threat to the enclave.
Control Flow Guard is part of the design
VBS enclave binaries also have additional build requirements.
Microsoft’s development documentation specifies Control Flow Guard instrumentation for enclave DLLs and uses /GUARD:MIXED in the linker configuration. Enclave DLLs also require integrity checking via /INTEGRITYCHECK.
The build model therefore becomes security-aware:
Compiler
|
+-- Code Generation
|
Linker
|
+-- /ENCLAVE
+-- /INTEGRITYCHECK
+-- /GUARD:MIXED
|
Signing
|
Deployment
This is important because an enclave is not just a normal DLL with a different loader flag.
The toolchain participates in the security boundary.
VEIID solves a subtle supply-chain problem
One particularly interesting part of the VBS enclave design is VEIID — the VBS Enclave Import ID binding utility.
The Microsoft documentation explains that VEIID updates import tables with known IDs for platform DLLs. The purpose is to prevent a malicious signed DLL with the same name as a platform DLL from being loaded.
This addresses a subtle but important problem:
Signed DLL
|
X
Expected dependency?
|
?
A valid signature does not automatically mean:
this is the exact dependency the enclave expected.
Security-sensitive software needs stronger identity binding.
That is the difference between:
"This file is signed."
and:
"This exact trusted component is being loaded."
Enclave memory is not normal process memory
Another critical property is memory isolation.
Microsoft notes that enclave memory is hidden from normal debuggers and protected from VTL0. VBS enclaves operate as VTL1 user-mode processes.
Simplified:
VTL0
+--------------------------------+
| Windows |
| |
| Host Application |
| Kernel |
+--------------------------------+
|
X
|
VTL1
+--------------------------------+
| VBS Enclave |
| |
| Sensitive Code |
| Sensitive Data |
+--------------------------------+
The point is not that VTL1 makes an enclave invulnerable.
It creates another protection domain.
The attacker controlling VTL0 should not automatically gain equivalent access to VTL1 enclave memory.
That is the security boundary.
The enclave API becomes part of the attack surface
There is a catch.
Every boundary needs an interface.
The application cannot simply manipulate enclave memory directly.
It must communicate with the enclave through supported mechanisms.
That means:
Host
|
| input
v
Enclave API
|
v
Sensitive operation
|
v
result
And therefore:
Host-controlled input
|
v
Enclave interface
|
v
Trusted code
becomes an attack surface.
If the enclave accepts:
length
pointer
algorithm
operation
buffer
metadata
then every one of these values must be treated as hostile.
The enclave may be isolated from the host.
It is not isolated from bad application design.
A buffer validation bug inside the enclave is still a security bug.
The boundary reduces the trusted computing base.
It does not eliminate software vulnerabilities.
Enclaves should be small
This leads to one of the most important design principles:
Do not put the whole application inside the enclave.
That defeats the purpose.
Bad design:
+--------------------------------+
| VBS Enclave |
| |
| Web Server |
| Database |
| Business Logic |
| Networking |
| Parser |
| Crypto |
| 200 dependencies |
+--------------------------------+
Better:
+--------------------------------+
| Host |
| |
| Web Server |
| Business Logic |
| Networking |
| Parsing |
+---------------+----------------+
|
v
+----------------+
| VBS Enclave |
| |
| Key handling |
| Signing |
| Decryption |
| Sensitive op |
+----------------+
The smaller the enclave, the smaller the trusted computing base.
That means fewer:
bugs
dependencies
APIs
parsers
memory operations
third-party libraries
A small enclave is easier to audit.
A giant enclave becomes another operating environment that nobody fully understands.
What should actually go into an enclave?
Not everything.
The strongest candidates are operations where exposure of the secret would be catastrophic.
For example:
Private key operations
Digital signing
Token protection
Credential handling
Sensitive cryptographic operations
Identity verification
High-value application secrets
The architecture could look like:
Application
|
| "Sign this hash"
v
VBS Enclave
|
| private key
|
v
Signature
|
v
Application
The application gets the signature.
It does not necessarily get the private key.
That is a dramatically better security boundary.
The same principle applies to authentication
Consider an application holding a long-lived credential.
Traditional:
Application
|
+-- API token
|
+-- memory
|
+-- attacker
Enclave-oriented design:
Application
|
| authentication request
v
Enclave
|
+-- protected credential
|
+-- validation
|
v
Result
Now an RCE in the host does not automatically equal disclosure of the raw credential.
Again, the host may still be fully compromised.
The objective is to make the compromise less valuable.
That is a much more realistic security goal.
This is the same philosophy behind modern hardware security
The industry has been moving toward smaller and smaller trust domains for years.
The progression looks roughly like:
Entire machine trusted
↓
Kernel trusted
↓
Hypervisor trusted
↓
Isolated security environment
↓
Hardware-backed root of trust
The idea is not to make software perfect.
It is to minimize the amount of software that must be trusted.
This is why Secure Boot, TPM, VBS, HVCI, PPL and enclave technologies belong to the same architectural conversation.
They are all attempts to answer:
Where is the smallest trustworthy place in the system?
VBS Enclaves are not the same as SGX
Microsoft supports multiple enclave technologies.
The current Windows documentation distinguishes:
VBS Enclaves
|
software-based
|
Windows Hypervisor
from:
Intel SGX
|
hardware-based
|
special CPU technology
Microsoft describes both as supported enclave technologies but makes clear that VBS Enclaves rely on the Windows hypervisor and do not require special enclave hardware.
This matters operationally.
You don’t need to design the application around a specialized CPU enclave instruction set.
The protection is integrated into the Windows virtualization architecture.
There is still a huge limitation
VBS Enclaves cannot magically protect secrets that must eventually leave the enclave.
Imagine:
Enclave
|
| plaintext secret
v
Host
You just destroyed much of the benefit.
The enclave should instead perform an operation without exposing the underlying secret:
Host
|
| encrypted input
v
Enclave
|
| protected key
|
| operation
v
Encrypted / signed result
|
v
Host
The architectural rule is:
Keep the secret inside the boundary for as long as possible.
That sounds obvious.
It is also where many enclave designs fail.
The host is still allowed to lie
Suppose the host says:
"Sign transaction A."
The enclave signs it.
But what if the host is malicious?
Maybe it asks the enclave to sign:
Transaction B
instead.
The enclave may protect the private key perfectly and still perform an unauthorized operation.
This means enclave security requires semantic authorization, not merely memory isolation.
The enclave needs to know:
What operation is allowed?
For whom?
Under what conditions?
On which data?
With which identity?
The boundary protects the secret.
The application protocol determines what the secret is allowed to do.
That is a much harder problem.
The threat model must explicitly assume host compromise
A proper design document should therefore start with:
Trusted:
Hypervisor
VBS security environment
Enclave code
Enclave signing chain
Required hardware security primitives
Untrusted:
Host application
Host memory
Host input
Host filesystem
Host network
Potentially compromised kernel
That produces a very different architecture.
Instead of:
Host = trusted
you get:
Host = potentially hostile coordinator
That is the mindset required to design meaningful enclave applications.
Windows is moving toward smaller trust domains
Microsoft’s current Windows 11 security documentation places VBS Enclaves alongside AppContainer, Windows Sandbox and other application-isolation technologies. The stated purpose of VBS Enclaves is specifically to protect application secrets against admin-level attacks.
That is a major conceptual change.
Traditional application security asks:
Can this process access the secret?
Modern confidential-computing design asks:
Can the process operate on the secret
without ever possessing the secret?
That is a much stronger question.
What happens when the administrator is compromised?
This is where VBS Enclaves become particularly interesting.
Assume:
Attacker
|
v
Administrator
|
v
Host process
Traditional application:
Administrator
|
v
Application memory
|
v
Secret
Enclave-based application:
Administrator
|
v
Host process
|
X
|
v
VBS Enclave
|
v
Secret
The administrator may control the host.
The enclave is designed to remain a separate security domain.
This does not mean the administrator becomes irrelevant.
It means the application can establish a security boundary that does not depend entirely on ordinary Windows administrator privileges.
That is precisely why Microsoft describes VBS Enclaves as protecting sensitive application workloads from admin-level attacks.
What happens when the kernel is compromised?
This is even more interesting.
Traditional:
Kernel compromise
|
v
Process memory
|
v
Secrets
VBS:
Kernel compromise
|
X
|
Hypervisor-protected environment
|
v
VBS security domain
The security model assumes the kernel can be compromised and places selected security assets below that boundary.
This does not make kernel exploits harmless.
It changes what a successful kernel exploit is allowed to reach.
That is the entire point.
The security boundary is moving downward
This is the real story behind VBS.
Security is moving from:
Application
↓
Kernel
toward:
Application
↓
Kernel
↓
Hypervisor
↓
Hardware
And sensitive application workloads can occupy an isolated region within that architecture:
Application
|
+-------------------+
| |
v v
Normal execution VBS Enclave
| |
v v
Windows Kernel Sensitive State
| |
+---------X---------+
|
Hypervisor
|
Hardware
That is a fundamentally different way of thinking about Windows security.
The uncomfortable conclusion
No operating system can promise that applications will never be compromised.
No kernel is bug-free.
No hypervisor is bug-free.
No signing infrastructure is perfect.
No security boundary is absolute.
The only sustainable strategy is to reduce the amount of trust placed in every layer.
That means:
Do not trust the application with every secret.
Do not trust the administrator with every secret.
Do not assume the kernel is invulnerable.
Do not make the security boundary larger than necessary.
VBS Enclaves are one implementation of that philosophy.
Their importance is not that they create an invincible container.
Their importance is that they let developers say:
This small piece of code and these few secrets are allowed to trust something that the rest of the application does not.
That is a much more powerful security model.
Practical architecture for a high-value application
For an application handling sensitive cryptographic material, a modern architecture could look like:
Internet
|
v
+---------------+
| Host Process |
| |
| API |
| Business |
| Logic |
+-------+-------+
|
Enclave API
|
v
+----------------------+
| VBS Enclave |
| |
| Private Key |
| Signing Logic |
| Credential Logic |
| Sensitive Operations |
+----------+-----------+
|
X
|
Protected Memory
|
Windows VBS
|
Hypervisor
|
Hardware
The host can crash.
The host can be exploited.
The host can be malicious.
The design goal is that the most valuable secrets do not automatically fall with it.
What administrators should verify
VBS Enclaves are an application architecture, but the underlying Windows security stack still matters.
At the platform level, verify:
Get-CimInstance Win32_DeviceGuard
Look at:
VirtualizationBasedSecurityStatus
SecurityServicesRunning
SecurityServicesConfigured
You should also verify that:
VBS
HVCI / Memory Integrity
Secure Boot
are actually running rather than assuming they are enabled because the hardware supports them.
Microsoft’s documentation provides separate configuration and verification guidance for Memory Integrity and VBS.
The distinction matters:
Supported
≠
Configured
≠
Running
Security engineers should always care about the third one.
The future of Windows security is not another antivirus checkbox
The interesting evolution of Windows security is happening underneath the traditional application model.
We already have:
Secure Boot
TPM
VBS
HVCI
PPL
Credential Guard
AppContainer
Windows Sandbox
Now add:
VBS Enclaves
The pattern becomes obvious.
Windows is progressively breaking the assumption that one compromised layer should automatically compromise everything above or below it.
That is the real purpose of modern isolation.
Not perfect prevention.
Compartmentalization of failure.
Final Takeaway
The most important change introduced by VBS Enclaves is not another Windows security feature.
It is a change in the question developers should ask.
Old question:
How do I keep my application from being compromised?
New question:
What happens to my most valuable secrets if my application is compromised anyway?
That second question is far more realistic.
A vulnerable application may still happen.
A malicious administrator may still happen.
A kernel exploit may still happen.
The security architecture should therefore assume that some boundaries will eventually fail.
The goal is to ensure that one failure does not collapse every other boundary at the same time.
VBS already changes the Windows trust model by placing selected security mechanisms in an environment protected by the hypervisor and by explicitly accounting for potential kernel compromise. VBS Enclaves take that philosophy into application architecture, creating a smaller trusted execution domain for sensitive code and data.
The direction is clear:
Old model
Trust the machine.
Trust the kernel.
Trust the process.
Modern model
Trust as little as possible.
Isolate what matters.
Assume compromise.
Protect the secrets anyway.
That is where Windows security is heading.
And the most interesting security boundary may no longer be between user mode and kernel mode.
It may be between the application you compromised and the secret you still cannot reach.
Technical Sources
- Microsoft Learn — Virtualization-based Security (VBS).
- Microsoft Learn — Virtualization-based security (VBS) Enclaves.
- Microsoft Learn — Secure Enclaves / Trusted Execution.
- Microsoft Learn — VBS Enclaves Development Guide.
- Microsoft Learn — Windows 11 Application Isolation.
- Microsoft Learn — VBS System Resource Protections and hypervisor-enforced boundaries.
🔐 Luki w protokole SMB: Czy Windows 11 jest odporny na EternalBlue i jego następców? 📌 Wprowadzenie Protokół SMB (Server Czytaj dalej
Dlaczego Windows 12 nie instaluje się i zatrzymuje na 0%, 20% lub 60% Instalacja systemu operacyjnego Windows 12 (w praktyce Czytaj dalej
