Immutability: The Cornerstone of Embedded Defense

Immutability means that something cannot be altered once it has been created or deployed. In cybersecurity, this property is highly desirable: if an attacker can’t modify a system, they can’t compromise it, at least not persistently.

But immutability isn’t binary. There are degrees of it, ranging from absolute (hardware-enforced) to soft, policy-based protections. Each level comes with its own trade-offs between security, flexibility, and maintainability.

At one extreme, you have ROMs or flash chips with hardware write protection, where nothing can change post-deployment. Perfect for secure bootloaders, but a nightmare for updates. At the other end of the spectrum, a regular Linux system with no write protections at all, where everything is mutable and thus vulnerable. Somewhere in the middle, you might find a SquashFS-based root filesystem: read-only by design, yet still replaceable if someone has physical access to the board or the bootloader.

Between those extremes lies a rich landscape of techniques: read-only mounts, filesystem-level immutability flags, overlay filesystems, and integrity verification tools, each balancing the need for protection with operational practicality.

In this article, we’ll explore the different shades of immutability in filesystems, their implementation strategies, their strengths and weaknesses, and how to work around immutability without breaking security.

The Spectrum of Filesystem Immutability

Immutability in filesystems isn’t a one-size-fits-all concept. There’s a spectrum of approaches, ranging from hard-wired protections that are physically impossible to bypass, down to software-enforced mechanisms that provide only modest resistance to tampering. To understand this landscape, it’s useful to consider the granularity of control:

Filesystem-level immutability applies to entire partitions or volumes. It’s simple to enforce but coarse: either everything is locked down, or nothing is.
Inode-level immutability operates on individual files or directories. It allows selective protection, offering fine-grained control.

Each technique along this spectrum comes with its own security guarantees, update strategies, and risks. There is also a large combination of strategies. Let’s walk through the most commonly used options.

1. Hard Immutability: Burned or Locked Media

Granularity: Device or image level
Enforcement: Hardware

The most rigid form of immutability is enforced at the hardware level. This includes:

Code stored in ROM or OTP (one-time programmable) memory
Flash devices with hardware write protection (e.g., WP pins tied to ground)
eMMC or NOR flash locked using secure boot fuses or control registers

✅ Pros:

Truly immutable once deployed
Immune to software-based tampering even with root access or physical attacks

❌ Cons:

Cannot be patched or updated without hardware replacement
Any mistake in the image is permanent

This is the gold standard for securing bootloaders, root-of-trust code, or fallback recovery environments, where updates are either unnecessary or intentionally prohibited.

2. SquashFS: Immutable by Design

Granularity: Filesystem-level
Enforcement: Filesystem format, no write primitive

SquashFS is a read-only filesystem commonly used for the rootfs in embedded Linux systems. It’s often referred to as “immutable by construction,” because once the image is built, it cannot be modified in place.

SquashFS supports optional compression, possibly hardware-assisted, which helps reduce storage usage. Compression can also be disabled to improve performance or simplify update workflows.

✅ Pros:

Read-only by design: no runtime writes possible
Compression (if used) reduces flash footprint
Suitable for cryptographic integrity verification

❌ Cons:

Cannot be updated in-place; requires full image replacement
Can be modified externally (e.g., via bootloader or debug interface), although this will be detected and handled

When combined with secure boot and signed updates, SquashFS strikes a solid balance between security and updatability. It is widely used in systems that require a stable, locked-down root filesystem with occasional over-the-air or field updates.

3. The immutable File Attribute (+i)

Granularity: Inode-level (file or directory)
Enforcement: Filesystem-level extended attributes (metadata)

On Linux systems using ext2/3/4, btrfs, or xfs, individual files and directories can be marked with the immutable attribute. This simple flag can provide strong, fine-grained immutability without changing filesystem layout or permissions.

When set (chattr +i), the immutable flag overrides traditional file permissions. Even if the file is marked as writable, it cannot be modified, renamed, deleted, or overwritten. Not even the root user or the file’s owner can change the contents: only root can clear the flag using chattr -i before any modification can be made.

On files:

All write operations are blocked
rm, mv, echo >>, and similar commands fail silently or with “Operation not permitted”

On directories:

No new files or subdirectories can be created
Existing entries (contained files) cannot be renamed or removed
The contents of the existing contained files can be altered (if not otherwise protected)

✅ Pros:

Independent of and stronger than standard file permissions
Provides strong protection against accidental or malicious changes
Easy to apply selectively

❌ Cons:

The flag can still be overridden by root, unless additional mechanisms (like LSMs or capabilities) are used
Not supported on every filesystem or platform
May confuse developers unfamiliar with the mechanism: the write protection does not appear in the file’s permission bits (e.g. with ls -l).

Despite its power and simplicity, the immutable flag is seldom used and frequently overlooked in system hardening. It’s an effective tool, particularly in embedded environments where key files should never change, but one that rarely makes it into standard deployment practices. Sad!

Note that there’s also a similar +a flag that enforces an append-only rule. It allows data to be added to a file but not removed or overwritten, making it particularly useful for protecting log files from tampering.

4. Read-Only Mounts

Granularity: Filesystem-level
Enforcement: Kernel runtime setting (soft)

Standard filesystems like ext4, btrfs, or xfs can be mounted with the ro (read-only) flag. The Linux kernel will reject all write operations to the mounted volume until it is explicitly remounted with rw.

✅ Pros:

Easy to implement
Supported on nearly all filesystems
Protects entire partitions from accidental or unauthorized writes

❌ Cons:

Remountable read-write by any process with root privileges
No fine-grained control: everything is either writable or not
Applications expecting write access (e.g., to /etc or /var) must be treated separately

This method is often used to protect the root filesystem, while placing writable data (logs, temporary files, configs) on separate partitions or overlays.

5. Unix-Style Permissions

Granularity: Inode-level (file or directory)
Enforcement: Filesystem and user privilege model (softest)

The most basic and widely known form of write protection in Unix-like systems is the standard read/write/execute (rwx) permission model. By removing write permissions from files or directories, administrators can prevent accidental changes, at least by non-privileged users.

✅ Pros:

Simple and universally supported
Easily understood and managed using standard tools
Works for both individual files and entire directory trees

❌ Cons:

Offers no protection against root and limited protection against users in the same group (if group permissions are not restricted).
Can be trivially bypassed by any process with elevated privileges
Often used more for convenience than for actual security

This mechanism is best described as soft immutability. It’s suitable for organizing access controls among users or processes, but should never be relied on for security boundaries, especially in embedded or production environments.

Managing Exceptions Without Breaking Security

Absolute immutability sounds ideal, but real systems need to change. Logs must be written, configurations occasionally updated, and software patched. The challenge is to allow these changes without compromising the underlying integrity of the system.

This section explores controlled methods to handle mutability without opening the floodgates, using temporary overrides, overlay filesystems, and update-safe patterns. Each of these techniques balances flexibility with containment.

1. Overlay Filesystems: Writable by Design

One of the most widely used strategies for enabling limited mutability on top of an immutable base is the overlay filesystem. This mechanism allows a read-only lower layer (e.g., a SquashFS rootfs) to be combined with a writable upper layer, often implemented using tmpfs (RAM). Under some circumstances, the upper layer can also be ext4 for persistence.

✅ Pros:

Keeps the original base untouched
Typically allows temporary changes, persistence being an option (risky!)
Compatible with existing tools and file layouts

❌ Cons:

Adds complexity to the filesystem layout
Persistent overlays can still be abused by malware
Requires careful management of update and rollback mechanisms

Typical use cases:

Kiosk systems that revert to a clean state on reboot
Devices that need writable /etc, /var, or /home
Development builds where mutability is temporarily allowed

For higher integrity, non-persistent overlays (e.g., tmpfs) ensure that any tampering is wiped out at reboot. For flexibility, persistent overlays let changes survive across boots, ideal for configuration or patch storage.

2. Temporarily Lifting Protections

Sometimes the system itself needs to perform updates or reconfiguration, and must therefore briefly become writable. This is a controlled breach of immutability with strict time and privilege boundaries.

Examples include:

Remounting the root filesystem as writable
Temporarily clearing the immutable flag

✅ Pros:

Compatible with update scripts and maintenance tools
Minimal change to existing system structure

❌ Cons:

Introduces a window of vulnerability: an attack during this period may succeed
Relies on scripts or human operators restoring protections correctly. Caution is needed to avoid interference from concurrent processes.
Logging and rollback are often not built-in

To reduce risk:

Require authenticated updates
Use sealed maintenance modes (e.g., via recovery partition)
Log every modification and validate system state post-update

3. Selective Mutability by Design

In many systems, only certain parts need to be writable: logs, temporary files, or configuration overlays. The rest of the system should remain immutable. This leads to a partitioned mutability strategy, where specific mount points or directories are granted write access.

Here is a common layout:

/ (rootfs) → SquashFS or ext4 mounted read-only 
/var/log → ext4, writable 
/etc → overlayfs with tmpfs upper 
/home → ext4 or btrfs, writable

This model allows:

Controlled write access only where needed
Read-only enforcement in sensitive system areas
Easier monitoring of changes (e.g., diffing overlay layers)

✅ Flexibility through composition:
One of the strengths of Linux-based systems is that these immutability techniques can be layered and composed. With the right design, this enables fine-grained immutability, secure update paths, and runtime flexibility without compromising the system’s overall integrity.

When Immutability Isn’t Enough: Falling Back on Integrity Checks

Even in systems designed to be immutable, some degree of change sometimes happens, whether by design or by subversion. In such cases, integrity verification becomes the next best defense: it doesn’t prevent modification, but it ensures you notice when something has changed.

When full immutability isn’t practical, integrity checking helps maintain trust in the system by detecting deviations before they escalate into compromise. Some filesystems have built-in integrity checks, and many other solutions exist (dm-verity, basic checksums, etc).

Practical Considerations for Real-World Projects

In practice, adopting immutability isn’t just about picking technical tools. It requires architecture planning, design-time trade-offs, and tooling discipline across your development pipeline. Here are a few key points to keep in mind:

Development vs Production Builds

Immutability is great for security but it gets in the way of development. Developers need fast iteration, writable filesystems, and minimal friction. That’s why we strongly recommend provisioning at least two target profiles:

*-dev builds: writable, debug-friendly, no overlay, no +i, logs accessible
*-prod builds: hardened, immutable, overlays or tmpfs where needed, permissions locked down. Must be the validation target.

This separation keeps your product secure without frustrating your team, and avoids the temptation to disable protections “just for testing” on production hardware.

Immutability Affects Your Update Strategy

Your choice of update mechanism must align with your immutability model. Some update frameworks like RAUC or Mender operate at the image or filesystem level, ideal for fully read-only root files. Others, like package-based updates (e.g. opkg, rpm, apt), assume writable files and modify them in place.

If your system enforces immutability at the filesystem level, package managers may fail silently or corrupt the image.

🔗 More on this in our dedicated article: OTA Update Tools: Find The Perfect Fit For Your Application

In short: plan ahead. Immutability is not just a technical detail: it impacts your whole architecture and deployment pipeline.

Cybersecurity Is Not a Post-Process

At Embedded Expertise, this is a golden rule we repeat often.

Cybersecurity must be built-in, not bolted on later. Immutability is not a corner case or an optional extra. It’s a global design constraint that needs to be accounted for from the beginning. It affects:

Partition layout
Update method
Development strategy
Recovery flow
Application architecture

Treating immutability as a first-class constraint will lead to better architecture decisions—and systems that are secure by design, not by patchwork.

Need Help Making Your System Immutable?

At Embedded Expertise, we help embedded teams design secure, maintainable systems from the ground up. Whether you’re building a hardened Linux device, selecting an OTA update strategy, or retrofitting security into an existing platform, we bring practical solutions grounded in real-world experience.

From immutable root filesystems to secure update pipelines, we’ve designed it, deployed it, and debugged it.

Need a design review? A security audit? A second opinion? Get in touch with us! We’re here to help you make the right trade-offs, without compromising your product or your timeline.

Key Takeaways

Immutability is one of the most effective defenses available in embedded cybersecurity. It limits the attack surface, prevents persistent compromise, and provides a predictable system state. But as we’ve seen, immutability is not binary: it exists on a spectrum, from fully locked-down ROM to soft enforcement through file permissions.

Hard immutability offers maximum security but no flexibility.
Filesystem-level protections (like SquashFS or read-only mounts) strike a balance.
Inode-level flags (like +i) provide fine-grained control.
Soft models (like Unix permissions) offer convenience but little real security.
And when changes must be allowed, overlays and integrity checks let us control or detect them.

In practice, the strongest designs combine multiple techniques: immutable root filesystems, read-only mounts, protected config files, controlled update paths, and fallback integrity mechanisms.

✅ The goal isn’t to eliminate change, it’s to control it.

Design your systems to be immutable by default. Then make carefully scoped exceptions documented, logged, and ideally reversible. In doing so, you’ll raise the bar for attackers while still allowing your systems to evolve safely.