Defense in Depth: 10 Practical Lessons
A single bad memory access was enough to permanently lock an entire system based on a popular SoC. Here are ten practical lessons from investigating and mitigating that failure.
The previous two articles started from a very concrete failure on the i.MX8M Plus: a single access to certain physical addresses can lock the complete SoC until an external reset or power cycle.
The first article used that failure to illustrate why perimeter protection is not enough. The second looked at mitigation: restricting access to the weak point, containing the propagation, providing recovery oustside the failure domain.
This third article is the condensed version.
No architecture discussion, no MPU tutorial, no catalogue of registers. Just ten practical lessons directly derived from what the black-hole investigation taught us.
1. "Reserved" does not mean protected
A region marked Reserved in a reference manual is an instruction to software, not necessarily an access-control mechanism. In our case, accessing such a region did not fail cleanly, it locked the complete SoC.
Whenever a reserved or undocumented area matters to product robustness, do not assume what the hardware will do. Verify the actual behavior and decide whether an explicit protection is required.
2. Application correctness is not enough
“The application never accesses that address” is not a sufficient countermeasure.
That reasoning assumes that the software will always behave exactly as designed. Bugs, corrupted pointers, malformed DMA descriptors, future code changes or compromised components can all violate that assumption.
Application correctness is necessary, but defense in depth requires lower layers to limit the consequences when it fails.
3. Prefer whitelisting where possible
Once several black holes were discovered, protecting each one individually became increasingly unattractive. A blacklist depends on knowing every dangerous region and being able to describe each one with the available protection hardware
Where practical, it is safer to start from no access and explicitly allow only the memories and peripherals a component actually needs. This is particularly relevant on the Cortex-M7, where MPU regions are limited and alignment constraints make irregular exclusions expensive.
4. Check what each protection mechanism actually covers
The presence of a security block does not mean that every transaction passes through it. On the i.MX8M Plus, the RDC protects a defined set of resources, and some of the dangerous regions we found are simply outside that set.
The same verification should be made for MMUs, MPUs, TrustZone controllers and peripheral firewalls: identify exactly which masters, targets and address ranges they can constrain before relying on them.
5. Vendor defaults are not product settings
A vendor BSP or SDK is designed to make development convenient and to expose the capabilities of the device. That may mean permissive MPU settings, accessible debug interfaces, /dev/mem, recovery paths and other facilities that are entirely reasonable during bring-up.
None of those defaults should automatically become production policy. Kernel configuration, MPU setup, debug access and boot options all deserve an explicit product-level review.
6. Remove production access paths you do not need
If JTAG is not required in the field, do not merely leave the connector unpopulated. A determined attacker can still reach test pads, vias or pins. Disable the capability at silicon level when the device provides that option, and include the fuse programming in the manufacturing process.
Apply the same reasoning to other development paths, including boot ROM download mechanisms such as the one used by uuu: if the deployed product does not need them, remove them.
7. Security and safety can meet at fault containment
The black-hole failure does not care whether the bad transaction was generated by an attacker or by a software defect. The consequence is the same.
Protection mechanisms that prevent one component from reaching unrelated hardware therefore improve both security and robustness. MPU restrictions, MMU mappings, privilege separation and bus-level access control should be reviewed not only as security features, but also as containment mechanisms for accidental faults.
8. Recovery must be outside the failure domain
The on-chip watchdog did not recover this failure because it was affected by the same system-wide lockup. That is a useful architectural lesson: a recovery mechanism is only independent if it survives the failure it is expected to recover from.
An external watchdog at board level can turn a permanent lockup into a temporary outage followed by a reset. For critical failure modes, test the real recovery path against the real fault rather than assuming that an internal watchdog is sufficient.
9. Accept that some residual risk will remain
Not every master can necessarily be constrained, not every dangerous address may sit behind a usable firewall, and finite MPU resources can force compromises. At some point, further protection becomes impractical or impossible. The remaining failure mode then has to be evaluated against the product requirements. If an external reset makes it acceptable, document that mitigation. If even a short outage is unacceptable, the answer may be a different partitioning, additional hardware supervision, or a different SoC.
Security has something in common with human health: it is not a permanent state you reach once and keep forever. It is a temporary balance that has to be maintained.
Software changes, configurations drift, new interfaces appear, dependencies evolve, and assumptions that were valid at product launch may no longer hold a few years later.
10. Revisit the analysis over the product lifetime
The protection strategy is not finished when the first production image ships. Kernel configurations change, BSPs are updated, firmware grows, new peripherals are enabled and manufacturing or recovery requirements evolve. Any of those changes can reopen an access path that was previously closed.
The list of allowed resources, debug interfaces, boot paths and recovery assumptions should therefore be reviewed whenever the platform changes materially.
This Failure Case Is Not the Point
The black-hole failure is only one illustration of defense in depth, with the problem materializing at the deepest layer we can encounter: the silicon itself.
But the same principle applies across the whole embedded stack, from hardware to application, through boot, OS configuration and middleware.
The real objective is not to design around one known weakness. It is to build a system that remains robust when an assumption fails, including when the underlying weakness was not known in advance.
That is why deep platform knowledge matters. It lets architecture replace assumptions with deliberate boundaries and protections.
The black hole is the example. The resilient architecture is the point.
Enjoyed this article?
Embedded Notes is an occasional, curated selection of similar content, delivered to you by email. No strings attached, no marketing noise.
Related resources
- Defense in Depth, When the Unexpected Happens
- Defense in Depth in Practice: Prevent, Contain, Recover
- Immutability: The Cornerstone of Embedded Defense
- Watchdogs: One Concept, Many Meanings
- Platform Architecture service · en | fr
- Embedded Cybersecurity · en | fr
- Training · en | fr