Salesforce 2019: The Permissions Bug That Became a Global Service Shutdown
Salesforce’s May 2019 “Permageddon” began when a database script changed permissions in Pardot-related orgs, prompting Salesforce to restrict access broadly while it reconstructed authorization state.
A database script modified authorization metadata instead of ordinary business data
On May 17, 2019, Salesforce discovered that a database script associated with Pardot had changed user profiles and permission sets in affected customer organizations, granting broader access than intended.[1] This was not a typical data-corruption incident. The records at risk described who was allowed to see and do what inside customer environments. Authorization metadata is security-critical state: when it becomes unreliable, the provider cannot safely assume that ordinary application access should continue. Salesforce responded by restricting access while it identified affected organizations and rebuilt permissions.
Permissions are production data with security semantics
An application can keep its business records intact and still be unsafe if the access-control layer no longer describes the intended policy.
Salesforce chose containment even though containment created an outage
To protect customers, Salesforce blocked access to instances containing affected organizations and temporarily reduced permissions.[1] That decision spread disruption to some customers that were not directly affected by the bad permission change but shared infrastructure with organizations that were. This was a deliberate tradeoff: availability was sacrificed to prevent possible unauthorized access. In security-sensitive failures, the safest recovery state may be “deny until known good,” even when that turns a localized metadata defect into a broad service interruption.
Fail closed is operationally painful but often correct
When authorization integrity is uncertain, continuing to serve normally risks a confidentiality breach that can be harder to reverse than downtime.
The blast radius reflected the relationship between tenants and shared instances
Salesforce explained that only organizations using or previously using Pardot were subject to the permission modification, yet it initially blocked all access to instances that contained affected orgs.[1] Multi-tenant platforms therefore have two different blast radii: the logical set of corrupted tenants and the operational set of infrastructure that must be isolated to protect them. Incident tooling needs enough tenant-level precision to shrink the second set toward the first as quickly as possible.
Tenant isolation affects recovery speed
The more precisely a provider can identify and quarantine affected authorization state, the less unrelated customer activity must be suspended during investigation.
Restoring permissions was harder than restoring ordinary records
Authorization configuration is highly customized. Profiles and permission sets encode years of customer decisions, and Salesforce could not safely replace them with one universal default.[2] Administrators needed mechanisms to restore or reconstruct their prior settings, including from sandbox copies where available. This makes authorization incidents different from simple database rollback: the provider must recover not just a record version but the customer-specific policy represented by that version.
Policy state needs backup and audit history
Organizations should be able to answer what permissions existed before a change, who changed them and how to return to that known-good state.
Salesforce gradually restored access as it separated affected from unaffected orgs
Once Salesforce isolated the affected organizations, it restored full access for unaffected customers and administrator access for affected orgs so recovery work could continue.[1] The phased restoration demonstrates a useful incident pattern: first choose a conservative global containment state, then reduce scope as evidence improves. This avoids making early optimistic assumptions while still minimizing unnecessary downtime once the provider can prove a tenant is safe.
The incident exposed how automation can turn metadata changes into security events
Profiles, permission sets and related authorization constructs are designed to centralize access policy in Salesforce.[3] That centralization is powerful but increases the importance of scripts and migrations that touch those objects. Bulk automation should use typed targets, tenant scoping, dry runs, change previews and invariant checks such as “this operation must not increase effective privileges.” Authorization changes deserve stronger review than ordinary data transformations because mistakes can silently widen access.
Modern least-privilege guidance reduces the consequences of bad permission changes
Salesforce security guidance increasingly favors permission sets, least privilege and deliberate access management rather than broad profile grants.[4] Least privilege cannot prevent a platform bug from changing permissions, but it reduces the normal amount of authority in the system and makes anomalous expansion easier to detect. Audit logs and configuration history likewise provide evidence needed to reconstruct intended state after an incident.
Why Salesforce 2019 belongs in software-failure history
Salesforce’s 2019 permissions incident belongs in software history because it shows that authorization metadata can be as operationally dangerous as lost data. A database script broadened access in Pardot-related organizations, and the provider responded by shutting down access broadly enough to guarantee safety while it reconstructed permission state. The lasting lesson is that access control is not static configuration sitting outside the application. It is critical live data that needs typed automation, backups, auditability, invariant checks and a recovery plan designed around the possibility that the permission system itself becomes untrustworthy.[5] The broader reliability principle is to make recovery a first-class production path. Teams should rehearse the exact operations they expect to use during failure, verify that monitoring and administrative tooling survive the same dependency loss, and measure how quickly backlogs and replacement capacity return to steady state. A recovery plan that exists only as documentation has not yet been tested as software.
Works Cited
- 01
- 02Salesforce Help — Root Cause Update for May 2019 Permission Incident help.salesforce.com
- 03Salesforce Help — Permission Sets help.salesforce.com
- 04Salesforce Architects — Well-Architected: Trusted Security architect.salesforce.com
- 05Salesforce Trust — System Status and Security trust.salesforce.com
CodeHistory is a living archive. Citations document the evidence used for this edition; later evidence may refine the account.