Lessons Learned from Recent Software Supply Chain Attacks

How to Protect Your Projects

On September 8, 2025, a single phishing email compromised 20 of the most widely-used JavaScript packages and later expanded to include duckdb, turning 2.6 billion plus weekly downloads into cryptocurrency theft tools. Eight days later, the "Shai Hulud" campaign transformed over 180 NPM packages, including those from cybersecurity giant CrowdStrike, into a self-propagating malware network.

These weren't nation-state attacks or sophisticated zero-days. They were simple social engineering campaigns that succeeded because our software supply chain security model has a fundamental flaw: we've optimized for convenience over verification.

The attacks revealed that every technical safeguard we've built; package signing, two-factor authentication, automated security scans; becomes irrelevant when attackers target the humans who control legitimate publishing credentials. When the publisher *is* the attacker, our security systems work perfectly to distribute malware.

Software supply chain attacks, far from being a recent phenomenon, have been a persistent and evolving threat for years. What we're witnessing now, however, marks a clear shift into a "new normal" where these sophisticated compromises are not just isolated incidents but a fundamental aspect of the cybersecurity landscape.

When Trust Becomes the Attack Vector

The "qix" and “duckdb_admin” incidents weren't sophisticated. Just a convincing phishing email sent to a maintainer of popular NPM packages.

But here's what makes this attack terrifying: it worked because our security systems functioned perfectly. The malicious updates were properly signed by the legitimate maintainer (who had been compromised). Two-factor authentication was enabled and working. The packages passed all automated security scans because the crypto-stealer was designed to look like normal code.

Every security measure we've built to protect the software supply chain became irrelevant the moment an attacker gained legitimate access.

The Evolution: From Simple Theft to Self-Propagating Malware

Eight days later, the "Shai Hulud" campaign proved that September 8th wasn't a fluke. It was a proof of concept. This time, attackers didn't just steal data; they turned the npm ecosystem into a self-replicating malware network.

The malware was disturbingly clever:

• It used legitimate tools like trufflehog to discover additional secrets in compromised environments
• It created GitHub Actions that continued stealing data long after the initial infection
• It modified package.json files and republished trojanized versions using stolen credentials
• Each compromised package became a vector for infecting its dependents, creating exponential growth

Even CrowdStrike (a cybersecurity company whose entire business is detecting threats) found their packages compromised.

The Uncomfortable Truth About Our Current Defenses

Here's what these attacks reveal: our current security model has a fundamental flaw. We've optimized for convenience and ease of use, but we've created a house of cards where compromising any single maintainer can cascade into ecosystem-wide damage.

Why our existing protections failed:

• Package signing verified the publisher's identity, but the attacker was the legitimate publisher
• Two-factor authentication protected accounts from credential stuffing, but not from credential theft after successful phishing
• Automated security scans missed malicious code designed to masquerade as normal functionality
• Dependency scanning couldn't detect threats hidden in trusted, widely-used packages

We need to accept an uncomfortable reality: in a world where attackers target humans rather than systems, no single security measure is sufficient.

Building Defenses That Assume Compromise

The solution isn't better locks. It's assuming the locks will eventually be picked. We need layered defenses where each component catches what the others miss, creating multiple points where attacks can be detected and stopped.

Dependency Pinning: Breaking the Infection Chain

"Shai Hulud" propagated by updating dependency versions, pulling compromised packages into clean environments. Strict pinning stops this cascade:

Essential pinning strategies:

• Exact versions: Lock to specific versions (or even SHAs, if available), not ranges
• SHA-pinned actions: Use commit hashes instead of moveable tags
• Human Review: All changes to SHAs should be reviewed by a human (fully automated merges with Dependabot or similar are highly discouraged)
• Shared workflows: Centralize CI/CD logic with strict review processes

Example: actions/checkout@v5 tags can be moved to malicious commits or by a branch with the same name, but actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 cannot be altered without detection.

Shared workflows create another barrier, where attackers must compromise heavily-protected central repositories instead of individual project workflows. This is also the recommended approach to achieve SLSA V1 Level 3 Build. SLSA (Supply-chain Levels for Software Artifacts) is a security framework, a set of standards and controls, for ensuring the integrity of software artifacts throughout the supply chain. It aims to prevent tampering and improve the security of software from source code to deployment.

Hardware-Backed Signing: When Credentials Aren't Enough

The "qix" attack succeeded because compromising a maintainer's account gave attackers everything they needed to publish malicious packages. Hardware-backed signing changes this equation by requiring physical hardware presence for signing operations.

Think of it like this: even if attackers steal your password and bypass your 2FA, they still can't sign packages without physically possessing your signing device. The cryptographic keys never leave tamper-resistant hardware, and each signature includes proof that it came from your specific device.

This isn't just theoretical. It's how the most security-critical systems already work. Code signing for iOS apps, firmware updates for medical devices, and financial transaction processing all rely on hardware security modules because the stakes are too high for software-only protection. 

Open-source package ecosystems are lagging. PyPI briefly supported package signing but removed it in May 2023. For npm, a 2013 pull request drew two months of sporadic discussion before being closed in 2015 with a link to a GitHub discussion. That discussion started in Jan 2015, hasn’t seen a comment since 2018, and remains open as of the publication of this blog. 

The challenge isn't technical. It's practical. Hardware security modules (HSMs) can be inexpensive. Options like Yubikeys, Nitrokeys and SoloKeys are great examples that can be used for package signing, committing and uploading code to software forges, and as a passkey or hardware 2FA (Second-Factor Authentication) for most modern applications. Trusted Platform Modules or TPM-based solutions and cloud HSM services are also making hardware-backed signing accessible to a broader range of developers. These can be used in combination with other methods discussed.

Reproducible Builds: Proving What You Published

Hardware-backed signing proves who published a package, but what if the build process itself is compromised? This is where reproducible builds become crucial: they prove that the published package actually contains what the source code claims.

When multiple parties can independently rebuild the same binary from source code, any discrepancy immediately reveals tampering. If an attacker modifies build scripts or injects malicious code during compilation, reproducible builds will catch the inconsistency.

Consider how this would have changed the "Shai Hulud" attack. If the malware had targeted build pipelines instead of source code, reproducible builds would have immediately flagged packages where independent rebuilds produced different results.

While this existed in the v0.1 SLSA Build L4 specification, it was later removed in the V1 specification, and made an optional add-on; our position is that it should remain mandatory.

More and more distributions have joined the Reproducible Builds Project, which aims to create an independently verifiable path from source to binary code. At this time, most of these distributions are not Full-Source Bootstrapped, leading to the potential of compiler backdoors.

This is a subset of the table from the StageX repository. Disclosure, I am a StageX maintainer.

Code Review: The Human Element That Machines Can't Replace

Automated tools excel at finding known vulnerability patterns, but they struggle with context and intent. Human reviewers can spot the subtle anomalies that indicate sophisticated attacks; changes that are technically valid but don't align with stated purposes, or modifications that seem out of character for a particular maintainer.

In the "Shai Hulud" attack, the malware made seemingly innocent changes to package.json files. Automated tools might flag these as low-risk metadata updates, but a human reviewer might question why a graphics utility suddenly needs network access or why a logging library is adding new dependencies.

Developers often integrate (and subsequently update) third-party dependencies without formal security reviews by either the security team or themselves. It's crucial to treat and review this code with the same rigor as any other code introduced by the team. We must avoid adding code without fully grasping its implications. While there are significant costs associated with such thorough reviews, these are far outweighed by the potential reputational and financial repercussions of neglecting them.

Why "Good Enough" Security Isn't Good Enough

You might be thinking: "We already have Trusted Publishers and integrity checks. Isn't that sufficient?" The September attacks prove it's not.

Trusted Publishers improve security significantly by providing verifiable provenance and reducing reliance on long-lived credentials. But they have a fundamental limitation: they only secure the final publishing step. They can't prevent malicious code committed to repositories, protect against compromised maintainer accounts, or detect subtle modifications that happen within legitimate workflows.

Think of Trusted Publishers as a high-quality lock on your front door. Essential, but useless if an attacker is already inside your house.

Defense Against Domain Resurrection

While npm was dealing with sophisticated attacks, PyPI was actively addressing threats like domain resurrection; where expired domains associated with legitimate packages are re-registered by attackers. Their work offers a blueprint for robust security.

PyPI's approach includes automated monitoring for domain expirations, preemptive notifications to maintainers, and suspicious registration alerts when previously legitimate domains change hands. These measures help prevent attacks or mitigate their impact.

While this is definitely a step in the right direction. There is still more that can be done. Increased transparency by registrars, backed by cryptographic provenance and zero-knowledge proofs could provide a robust framework for preventing these attacks and the board class of domain takeovers.

The Stakes Keep Rising

The September 2025 attacks weren't anomalies. They were previews of our new normal. As software becomes increasingly interconnected and attackers grow more sophisticated, the potential impact of supply chain compromises will only increase.

We're not just talking about stolen cryptocurrency or exposed API keys anymore. Modern applications control everything from medical devices to industrial control systems. A successful supply chain attack could affect not just developer productivity, but human safety, production, travel and critical infrastructure.

The Choice We Face

We stand at a crossroads. We can continue with our current security model (convenient, familiar, and fundamentally inadequate), or we can accept that the threat landscape has changed and our defenses must evolve accordingly.

The cost of better security is real: hardware requirements, more complex workflows, additional review processes. But the cost of doing nothing is catastrophic: an ecosystem where any successful phishing email can cascade into millions of compromised applications.

The September attacks taught us that our greatest strength (the interconnected, reusable nature of modern software) has become our greatest vulnerability. It's time to build defenses worthy of that reality.

Because the next time attackers target our supply chain, 2.6 billion downloads might just be the beginning.

‍