Unlock the ability of scale-across with Cisco
Each few years, a brand new class of workload arrives that breaks the assumptions of the earlier era and forces us to rethink not simply the structure, however the underlying physics of how bits transfer. Generative AI is one such second.
What makes this inflection level totally different is that we’re not simply asking networks to hold extra site visitors. We’re asking them to maintain a tightly synchronized, latency-sensitive compute atmosphere that spans a number of information facilities separated by tens or lots of of kilometers. That’s a basically new downside, and fixing it requires co-innovation throughout silicon, methods, and optics in methods the {industry} hasn’t tried earlier than.
Extra GPUs, extra intelligence
There’s a easy and profound reality on the coronary heart of the AI period: extra GPUs unlock extra intelligence. Each important leap in AI functionality over the previous six years—from language fashions that would full a sentence to methods that may purpose, code, and create—has been pushed by coaching on bigger clusters of GPUs. Scaling from lots of of GPUs to lots of of 1000’s has been instrumental in getting us right here.
However that trajectory has run into a tough bodily wall: energy. A single high-performance GPU attracts 700 watts (W) or extra at full load. A rack of GPU servers attracts 80 to 150 kilowatts (kW). A coaching cluster massive sufficient to develop a frontier AI mannequin can eat 10 to fifteen megawatts (MW), roughly equal to a small city’s electrical energy demand. And essentially the most superior fashions being educated in the present day require clusters that strategy or exceed 100 MW at a single web site, representing 60,000 to 70,000 GPUs or extra.
At this scale, energy has develop into the binding constraint. Energy availability and price in densely populated areas, mixed with the sheer magnitude of electrical energy required, implies that the most important AI coaching clusters have outgrown what any single facility can assist. Information facilities are migrating to less-populated areas with cheaper power, making interconnection of GPUs throughout information facilities a prerequisite. When the GPUs wanted to coach the subsequent era of AI are unfold throughout two or extra websites, the community connecting them should carry out as in the event that they have been in the identical room. This is why scale-across exists.
The bandwidth hierarchy: From DCI to scale-across
To know this new problem, it helps to hint how information heart bandwidth necessities have advanced. Every era has been extra demanding by orders of magnitude.
Conventional information heart interconnect (DCI) set the baseline. DCI joins information facilities to different information facilities and finish customers over wide-area networks. It was constructed for redundancy, geographic attain, and enterprise workload distribution.
Entrance-end networks emerged subsequent to deal with site visitors between customers, purposes, and cloud companies—video streaming, social media, cloud-native purposes—at roughly 7x the bandwidth of DCI.
The true step change, scale-up networks, emerged with AI. As information facilities pivoted from general-purpose compute to AI powerhouses, customary servers gave strategy to GPUs and specialised accelerators. Inside a rack, these gadgets are interconnected in scale-up domains at roughly 504x the bandwidth of DCI—related by high-speed copper at 100 to 200 Gbps per lane throughout distances of as much as 3 meters (m), showing to the software program stack as a single logical compute unit.
Scale-out networks then prolonged the AI material throughout a complete information heart, connecting racks of GPUs at roughly 56x DCI bandwidth by high-speed Ethernet and InfiniBand switching materials. As soon as distances develop past just a few meters—spanning rack rows and information heart flooring at reaches of 100 meters to 2 kilometers (km)—copper can now not keep sign high quality at these speeds, and pluggableome . Because of this, applied sciences like co-packaged optics and linear pluggable optics emerged to deal with the ability and density penalties of deploying optics at this scale.
And now we arrive at scale-across, the frontier the place the physics get genuinely onerous.
Scale-across networking: The promise—and the problem
Scale-across is the reply to the geographically distributed GPU downside, and it’s not merely DCI with increased bandwidth. Conventional DCI connects CPUs throughout information facilities and to finish customers, dealing with many low-bandwidth, loss-tolerant, asynchronous flows that develop linearly. Scale-across connects GPUs and scale-out networks, carrying a small variety of extraordinarily high-bandwidth, loss-intolerant, synchronous, long-lived flows that can’t tolerate dropped packets or timing mismatches with out forcing a full restart of the AI job. And people flows are rising exponentially.
The size distinction alone is placing. Scale-across networks require someplace between 12,000 and 32,000 ports—and context makes clear why. A 100 MW information heart homes roughly 60,000 to 70,000 GPUs, every producing as much as 800 Gbps of back-end site visitors. Connecting these GPUs inside a facility already calls for 1000’s of high-speed ports; extending that cluster to a second web site—whereas preserving the deterministic-latency, lossless efficiency of a dwell AI coaching job—requires 1000’s extra coherent optical ports on the scale-across layer. By comparability, conventional DCI usually makes use of 1,000 to 2,000 ports to deal with the identical two services’ enterprise site visitors. Each use coherent optics over distances exceeding 10 km, and each require strong safety. However the scale, site visitors traits, and efficiency tolerances are in a completely totally different class.
Conventional lossless networks depend on reactive congestion management, which struggles over lengthy fiber distances as a result of the velocity of sunshine means roughly 100 MB of information is in flight on a 100 km hyperlink earlier than movement management may even reply, consuming almost half a contemporary change’s buffer for a single port and precedence. That’s the reason deep-buffered routers, not switches, are the precise instrument right here.
AI workloads, nevertheless, supply an essential benefit: they’re predictable sufficient to make proactive congestion management potential, orchestrating site visitors to keep away from congestion earlier than it happens. However hyperlink failures at scale are unpredictable, and after they occur, you additionally want reactive management with deep buffers to soak up the disruption with out forcing the complete job to roll again to a checkpoint and incurring extra expense.
That is the place silicon and coherent optics converge round a single crucial: reliability. On the scale of AI coaching, hyperlink failures are inevitable. A single safety breach or episode of packet loss can erase 1000’s of GPU-hours of labor. Finish-to-end hardware-based safety, deep buffering for failure restoration, and proactive congestion management at the moment are desk stakes. Reliability is key to Cisco converged AI infrastructure, embedded at each layer.
Energy because the defining constraint and alternative
Energy has develop into the lens by which each and every architectural determination in AI networking should be evaluated.
On the silicon degree, energy effectivity is the deciding issue between a router that’s viable for high-density AI scale-across and one which falls quick.
On the optics degree, the identical logic applies, however the energy problem compounds because the community grows. Pluggable coherent optics scale back energy consumption by eliminating transponders and related consumer optics and permitting direct router-to-router connectivity. Freed-up energy may be redirected to GPUs delivering compute efficiency. However coherent pluggables remedy solely a part of the issue. As scale-across deployments develop from 1000’s of coherent ports to tens of 1000’s, the fiber infrastructure connecting these information facilities should scale in parallel. Extra ports imply extra fiber connections, and extra fiber connections imply extra optical amplification capability alongside these routes. Every of these amplification websites consumes energy of its personal. The result’s a two-sided energy problem: effectivity good points inside the information heart on the router-optics interface should be matched by effectivity good points alongside the fiber plant that connects them. Discovering the precise steadiness between efficiency and energy at each level within the community is now a first-order engineering downside.
The implication is evident: scale-across can’t be designed by optimizing silicon and optics independently. They should be co-designed from the bottom up.
How Cisco is converging silicon and optics in scale-across options
At Cisco, we’ve got been constructing towards this convergence for years. The mixture of the Cisco Silicon One–powered routing methods and coherent optics portfolio gives an built-in strategy designed particularly for what scale-across calls for:
Cisco Silicon One: Cisco Silicon One P200 powers Cisco 8223and methods to an industry-leading 51.2 Tbps capability, tailor-made for distributed AI workloads. the anticipated forecast development of Cisco AI orders in fiscal This autumn 2026 to greater than 6 billion. methods converge routing and switching with spectacular energy effectivity, programmability, and safety, enabling hyperscalers, neoclouds, and sovereign clouds to confidently architect geographically distributed AI environments.
Determine 1.
Coherent modules: Cisco is the coherent market-share chief and pioneer in coherent pluggables. 400G ZR/ZR+ and 800G ZR/ZR+ coherent pluggables are already being deployed in scale-across networks, with over 750,000 400G DSP ports shipped and over 40,000 800G DSP ports shipped. The broad Cisco coherent pluggable portfolio helps the mature requirements outlined by OIF, OpenROADM, and OpenZR+ which have enabled the mass adoption of router-based optics.
Determine 2. Cisco QSFP-DD 800G ZR/ZR+ coherent pluggable
Determine 3. Cisco OSFP 800G ZR/ZR+ coherent pluggable
Open line methods: Cisco gives two choices for purchasers relying on the use case:
- The brand new Cisco Open Transport 3000 Collection open line system gives a multi-rail structure that permits a number of fiber pairs to function in parallel so it may possibly deal with multi-petabit site visitors over lengthy distances. It additionally helps each C-band and L-band wavelengths, optimizing energy, house, and scalability for scale-across networks.
- The Cisco NCS 1014 metro open line system gives enhanced optical visibility and management that allows coherent pluggable deployments at scale in metro scale-across use instances. This consists of built-in coherent probe, dynamic achieve equalization, OTDR, and spectral energy monitoring that simplify deploying and working coherent optics which can be disaggregated from line methods.
Collectively, these capabilities kind a scale-across portfolio purpose-built for the reliability, energy effectivity, and scalability that AI infrastructure operators require.
What’s subsequent for scale-across
The scale-across period remains to be early. Networks that may energy the subsequent era of AI intelligence should be co-designed, from the coherent DSP and photonic integration on the optical layer, by the silicon and its buffer structure, to the system-level thermal and energy envelope that determines what is definitely deployable at hyperscale.
At Cisco, that’s precisely how we’re approaching scale-across. The Cisco Silicon One adaptive methods and coherent optics portfolio are designed in shut collaboration internally and with our clients to meet the particular calls for of scale-across. As AI continues its exponential trajectory, these applied sciences would be the key to unlocking new ranges of intelligence and enabling the subsequent era of AI infrastructure.
Discover Cisco Silicon One, a scalable and programmable unified networking structure
Extra assets
