As inference becomes more distributed, smarter networking will keep GPU’s productive and AI infrastructure efficient.
Aria Networks – a relatively new player in the networking space – is taking aim at performance problems emerging as AI inference becomes more distributed.
For years, the data centre network was largely treated as the plumbing between expensive pieces of compute – this is becoming harder to sustain though as inference is changing the architecture – instead of keeping everything inside a single server, operators are increasingly separating workloads such as prefill and decode, that can push the model’s KV cache (effectively its working memory) across the network. Aria argue this turns inference into a distributed computing problem in which the network sits directly in the critical path – meaning a network problem can quickly become a compute problem.
When the network slows, GPUs wait
An AI accelerator does not need to be completely offline to become inefficient – if it’s waiting for data, cache or another part of a distributed workload, expensive compute capacity is sitting idle. Aria recently described an example from its own testing in which network congestion caused a GPU node to run cooler because the accelerator was waiting for data. They say its telemetry identified the network as the cause – whilst the example is Aria’s own, rather than an independently verified benchmark – the underlying infrastructure issue is broader: as more of the AI workload moves across the fabric, network performance increasingly influences how effectively accelerator capacity is used. This is particularly important for inference, where traffic patterns can change rapidly and distributed workloads can move large amounts of KV cache between systems.
AI networking needs more visibility
Aria is actively targeting this problem with its Deep Networking platform which combines switching hardware, network telemetry, host information and workload data into a single view. The platform is designed to correlate network behaviour with GPU state, NIC counters and AI workload telemetry rather than treating those systems separately – the company says its telemetry operates at 100 to 10,000 times the resolution of traditional tools which points to an important operational question for AI operators: whether conventional network monitoring can see the short-lived congestion and latency events that affect distributed inference.
When the network becomes part of the AI system
This is the bigger shift – power teams worry about whether the facility can supply enough electricity. Cooling teams worry about removing enough heat. Compute teams worry about accelerator utilisation – Networking increasingly sits across all three. The next generation of AI data centres will not just need faster networks, they will need networks that can see what the AI workload is doing, understand when performance is slipping and respond quickly enough to keep expensive compute productive – this is the infrastructure gap Aria are trying to fill.
