Rafay Systems targets the operating layer of the AI infrastructure boom

The AI infrastructure market is moving through a critical transition. The first phase was about acquiring graphics processing units and standing up capacity. The next phase is about turning that expensive hardware into a secure, reliable and profitable cloud service.

That is where Rafay Systems Inc. sees its opportunity.

In my recent conversation with Haseeb Budhani (pictured), co-founder and chief executive officer of Rafay Systems, we examined the operational pressure facing neoclouds, sovereign cloud providers, telecommunications companies and enterprises as they deploy increasingly large AI systems.

The demand is unprecedented. Providers are buying infrastructure at massive scale, often with customers already waiting for capacity. But buying GPUs does not create a cloud. Operators still need orchestration, networking, security, multitenancy, observability, auditing and a developer experience that lets customers consume infrastructure without a lengthy manual process.

The central issue is time to revenue. AI infrastructure is expensive, depreciates quickly and must begin generating returns as soon as possible.

“End of the day, the thing that matters most is faster time to market and a better user experience,” Budhani said. “And if you can deliver both of those things, everybody wins.”

I recently spoke with Budhani during an exclusive CUBE Conversation to discuss the software and operational requirements behind large-scale AI infrastructure, the rise of sovereign and neocloud providers and the pressure to turn GPU capacity into revenue. (* Disclosure below.) 

AI infrastructure must become a service

Budhani’s definition of a cloud is straightforward: Customers should be able to visit a portal or use an application programming interface, press a button and receive an AI service in a multitenant environment.

“My definition of a cloud is where you must be able to go to a cloud provider’s page, click a button, and get an AI use case in a multi-tenant fashion,” he said. “I don’t want to talk to anybody. If you can do that, you’re a cloud. If you can’t do that, you’re not a cloud. You’re custom infrastructure.”

That distinction matters because many emerging AI providers are still building the operational capabilities that Amazon Web Services Inc., Microsoft Corp. and Google LLC developed over more than a decade. The hyperscalers had thousands of engineers and years to build their control planes. Today’s AI clouds may have only months.

They cannot afford to rebuild every layer themselves.

Rafay’s role is to provide the operating software between the hardware and the customer experience. The company helps providers expose bare metal, Kubernetes environments, virtual machines, serverless services, open-source models and token-based offerings through a common platform.

“The right AI cloud in this day and age, they do all of these things,” Budhani said. “It’s not just about bare metal or not just about Kubernetes or VM or tokens. It’s everything.”

That breadth allows an AI cloud to serve multiple customer segments. A large model developer may want Kubernetes. Another customer may want bare metal. An enterprise developer may expect a serverless experience, while another organization may simply want to purchase and distribute tokens.

The economic model improves as providers move higher in the stack. Bare-metal capacity may generate predictable revenue, but managed services and token consumption can deliver better margins.

Sovereignty expands the opportunity

The growth of sovereign AI is also reshaping the cloud market. Countries and regions increasingly want local infrastructure that keeps data and computing resources closer to home.

“The big driver is sovereignty of compute, sovereignty of data,” Budhani said.

The first wave came from local model builders and agentic application developers. Enterprise customers are now following. Many companies may continue running traditional workloads in the public cloud while selecting a regional provider for AI.

These enterprises bring the expectations they developed through years of using hyperscale platforms. They want quotas, policies, auditability, attestation and security controls. They also want a simple consumption experience.

The regional provider that can deliver those capabilities has an opportunity to become a meaningful AI utility in its market.

Delivery becomes the differentiator

Software is only part of the equation. Budhani emphasized that Rafay’s most important investment has been its ability to help customers deploy services rapidly.

“The delivery muscle is the most important investment we made in this company,” he said. “It’s not the software. Of course, software is important. That’s the bar, right? You have to have that.”

Rafay increasingly becomes involved while customers are purchasing infrastructure, not after the hardware arrives. The company works with them to identify target customers, understand their requirements and prepare the operating environment before capacity comes online.

In one example, a customer wanted to demonstrate a working service and begin signing users within 10 days. Rafay assembled engineers from both companies to build a minimum viable deployment immediately.

That urgency is becoming normal. Deployment cycles that once took quarters are compressing into weeks or days.

“If they take the same amount of time as all these other players who had the time two years ago, three years ago, they’re going to fail,” Budhani said. “A couple of months is a very long time in this industry because customers are waiting.”

AI infrastructure is an ecosystem business

No single vendor can provide the entire AI stack. A deployment may include Nvidia Corp. GPUs, Dell Technologies Inc. servers, multiple networking suppliers, specialized storage, security platforms and systems integrators.

Supply constraints make this environment even more heterogeneous. A provider may plan to standardize on one vendor, only to discover that it cannot obtain enough switches, servers or memory. The software layer must accommodate what is available.

“This is a truly a play where four, five, six, seven vendors come together to make one AI cloud successful,” Budhani said.

That changes the go-to-market model. Rafay is not simply selling software and handing the customer support. Its teams often coordinate infrastructure partners and align the technical work required to get a service running.

Rafay spent years building for cloud-native operations before the AI infrastructure demand curve arrived. What looked like a specialized platform is now becoming foundational to a much larger market.

The winners in this cycle will not be determined solely by who owns the most GPUs. They will be the providers that can operationalize infrastructure, protect enterprise data, deliver a frictionless customer experience and monetize capacity faster than the competition.

Here’s my full conversation with Budhani:

(* Disclosure: Rafay Systems sponsored this segment of theCUBE. Neither Rafay Systems nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Image: SiliconANGLE/ChatGPT