Patching on the pace of discovery: full-fleet Community OS upgrades in days, not months

As AI accelerates vulnerability discovery, Cisco IT has lower fleet working system (OS) improve occasions from months to days, shrinking the window attackers have to use a identified flaw with out sacrificing uptime.
By integrating improve choices into our community stock supply of reality and enabling our Catalyst Middle, Nexus Dashboard, and Meraki dashboard implementations to hold out these choices, we at the moment are in a position to apply hardened releases to over 95% of our Cisco IT-managed units inside our service-level goal of seven days from availability.
Our problem: advisories outpaced remediation
Frontier AI fashions are discovering vulnerabilities quicker than the normal disclosure-and-patch mannequin can soak up, and the hole between disclosure and exploitation is shrinking from weeks to minutes. The Cisco Product Safety Incident Response Crew (PSIRT) responded with security-hardening releases on a daily cadence every month, with seven days of advance discover for these deliberate releases. Russ Smoak’s (VP, Data Safety) “Strengthening the Basis” weblog covers the main points. Constructing these fixes nonetheless takes expert human judgment, as Jonathan Felten (Vice President, Engineering) describes in his “Cryptography Not often Fails Due to the Math” weblog. Right here we cowl the ultimate stretch: deploying them.
Cisco IT runs the identical {hardware} and software program as our clients and our previous workflow handled every advisory as an remoted investigation. We spent invaluable time parsing stock sources of reality, backup configs, and dwell system states simply to reply one query: Are we susceptible?
At our scale, that method broke down:
- Quantity and variety: With greater than 60,000 units together with IOS-XE, IOS-XR, NX-OS, Utility Centric Infrastructure (ACI), and others, every OS sort wanted its personal evaluation method.
- Model sprawl: Advisories arrived quicker than we may remediate inside our service-level targets (SLOs), which may be as brief as three to seven days for a important subject. Every OS sort fragmented into many variations.
- Handbook execution: System-by-device scripts and hand-authored change requests tied remediation pace to engineer availability, not patch availability.
How AI-discovered vulnerabilities change the panorama
Hardened releases embody many fixes and don’t element every bug, however should nonetheless be deployed. The query is now not “Are we susceptible?” however fairly, “How briskly can we improve safely?” Leaders weighed the tradeoff: a quick, deliberate disruption to improve in opposition to an unplanned safety occasion that might value way more. Safety and community management agreed early that the second threat is extra impactful, they usually dedicated to upgrading by default. The problem then turned supply.
Our method: optimizing OS picture administration
Our method spans 4 layers: the supply of reality is leveraged for choices, Cisco controllers execute and confirm, logs of the method are recorded in Splunk Cloud Platform, and AI brokers cowl the gaps.
After a human units the golden picture, automation performs the improve:
- Upkeep and eligibility checks
- Change report creation, battle decision, and approvals
- Picture distribution
- Pre-checks
- Improve system
If any of those steps fail, the system improve won’t proceed.
Human judgment stays the place it issues most: overseeing the method and resolving points as they come up. Fewer engineers overseeing every of our elements can now successfully scale to supervise 1000’s of modifications occurring over just a few days and even hours.
The important thing necessities that allowed us to implement this technique:
- Correct supply of reality: We should function from a dependable, identified state to make correct choices. In Cisco IT’s case, we leveraged our configuration administration database (CMDB) as a result of it comprises our stock with extra system attribution, and can be utilized to determine what will get upgraded, when it is going to be upgraded, and in what order.
- Change administration: Upgrades proceed to observe our established change administration requirements, together with monitoring each change and figuring out dependencies and conflicts. Automation doesn’t substitute approvals. It helps upgrades transfer by means of our present change course of extra constantly. Adjustments that meet our documented standards for the standard change (low threat, minimal impression and repeatable) observe a pre-approved path outlined in our change administration coverage, in order that they don’t want case-by-case assessment.
- Assurance capabilities: Since automation drives this course of, we will need to have confidence in reporting the state of the community and the progress of upgrades. Automation could make choices primarily based on topology (as outlined within the single supply of reality) to schedule upgrades to keep away from outages. Improve progress is logged to Splunk Cloud Platform, the place we use this knowledge to create dashboards that determine compliance points blocking system upgrades. This allows our group to shortly resolve blockers and permit upgrades to proceed.
- Simplify success standards: We needed to evolve our mindset about how we measure success of upgrades. Success is extra closely weighted as to whether the system upgraded efficiently vs operational points that will come up. If we encounter an operational subject, it’s addressed by means of separate incidents and escalated, if vital, as an alternative of being handled as an improve failure.
As soon as these choices have been made, execution turns into a product functionality, not a customized undertaking. Software program Picture Administration (SWIM), accessible to clients in Catalyst Middle, distributes and prompts photos throughout 1000’s of units with built-in pre-checks, post-checks, and rollback. Nexus Dashboard and Meraki dashboard have related capabilities of their domains.
The place a tool will not be but managed by a controller, automation and agentic operations fill the hole. For instance, the Cisco Community IT group has leveraged an AI operations agent to improve 200 non-Catalyst Middle managed units inside three days of the improve bundle turning into accessible. This AI operations agent additionally accelerated deployment of upgrades to Id Providers Engine (ISE) and assists with day-to-day community operations.
Matching pace to blast radius
The method is identical all over the place – how briskly we transfer is dependent upon identified impacts.
On the edge, we go quick. Taking a single change, entry level, and even half a constructing offline is suitable, particularly exterior enterprise hours. Topology guidelines maintain the opposite half on-line. The place business-critical groups function across the clock, we coordinate change home windows with them instantly. All over the place else, upgrades run autonomously. Utilizing this technique has enabled us to improve greater than 30,000 units inside their service-level targets.
Within the knowledge heart, we transfer shortly, however don’t rush. Every Utility Centric Infrastructure (ACI) cloth change impacts way more workloads, so we take a extra cautious but nonetheless aggressive method. Prior to now, upgrading 1200 ACI cloth switches took six months. It now takes 5 weeks, a fivefold discount. Three modifications made the distinction:
- Higher code high quality: Excessive-quality releases from Cisco engineering allow us to improve confidently throughout enterprise hours.
- Sooner approvals: Change requests higher account for non-impacting modifications to community configuration gadgets, eradicating most approval delays.
- Automation: Utility Coverage Infrastructure Controller (APIC) upgrades switches in staggered teams so redundant pairs by no means go offline In-house validation replaces guide checks, evaluating endpoints, protocol neighbors, and interfaces in opposition to a pre-upgrade baseline earlier than the following group begins. Batches now attain 34 switches at a time.
Outcomes and what we realized
The measure that issues most is the final mile: how lengthy a tool stays uncovered after a repair is obtainable. Our imply time to remediate (MTTR) has dropped from months to days, and hardened launch except for NX-OS has been utilized inside our service-level targets. Being buyer zero is ingrained in our tradition at Cisco – we deploy and operationalize Cisco merchandise at a worldwide scale. The success of this buyer zero deployment means upgrades aren’t simply examined in a lab, they’re deployed to over 60,000 units together with:
- IOS-XE units, together with cloud-managed switches
- IOS-XR units
- NX-OS units (contains ACI materials and standalone)
- Entry factors: Catalyst & Meraki cloud-managed
- Meraki Hybrid Employee Bundles
- Adaptive Safety Equipment (ASA) and Firepower Risk Protection (FTD) Firewalls
- A number of world Id Providers Engine (ISE) clusters
- Nexus Dashboard controllers
- SD-WAN management elements
- Webex units
This was completed with minimal operational impacts attributable to hardened launch upgrades.
Many of the achieve got here from streamlining change administration processes and decreasing wait occasions. Moreover, Cisco controllers scale the implementation by deploying upgrades in parallel as an alternative of system by system. Engineers at the moment are in a position to focus extra of their time on exceptions. Each change is traceable in Splunk Cloud Platform, and fewer one-off OS variants imply the following launch lands even quicker.
If you’re going through the identical strain, here’s what labored for us:
- Agree on precedence first: Deal with hardened releases as must-deploy work.
- Get the CMDB proper: Producing an correct stock is a prerequisite, then layer OS requirements, change guidelines, and topology guidelines on high of it.
- Use the product functionality you have already got: Begin with SWIM in Catalyst Middle earlier than writing customized automation.
- Pre-approve what you repeat: Transfer well-understood, low-risk upgrades to plain modifications to permit for automated approvals. Make the most of AI operations brokers with human-approval guardrails to deal with processes not but automated.
- Match pace to blast radius: Go quick the place failure is contained. The place it’s not, transfer shortly, however don’t rush.
- Use the seven-day discover: Pre-stage improve bundles on controllers and units, carry out lab validation, and automate approvals when PSIRT publishes the know-how checklist.
What’s subsequent
Cisco PSIRT “Hardened Launch” advisories are launched on a schedule, however exploits are usually not. Cisco IT and safety groups at the moment are working collectively on the tougher case: a identified, extreme vulnerability that can’t wait weeks for an improve. Our aim is to mitigate and remediate inside hours to days.
- Discover and comprise: Our Assault Floor Administration program and platform will floor important exposures shortly. Till a patch lands, we are going to comprise every publicity with essentially the most exact management for that asset, equivalent to Cisco Stay Defend on community units, endpoint detection and response (EDR) on hosts, or focused community coverage, fairly than using blunt devices such community isolation.
- Make room to patch: We’re eradicating hurdles to vary home windows and establishing standing home windows throughout which groups can improve or patch. Routine upgrades transfer towards an ‘inform’ mannequin, the place infrastructure and utility groups get visibility into what’s altering as an alternative of approving every change when no impression is anticipated.
- Design for patching: Each utility has totally different resilience wants. We’re defining Cisco IT-validated reference designs matched to every degree of criticality. Groups that implement these designs know their functions will face up to routine patching.
AI has collapsed the time between discovery and exploitation. Cisco IT’s evolving practices are designed to maintain tempo by minimizing change hurdles, automating upgrades by means of enterprise processes on Cisco controllers, and integrating human ruled AI brokers into our engineering workflows. That’s digital resilience in follow – patching close to the pace of discovery whereas the enterprise retains operating with minimal disruption.
Shifting this quick will floor new friction earlier than it removes the previous. We plan to share what we study as we go.



