Innovations in AI infrastructure: building custom AI accelerators
Broadcom brings more than 10 years of AI engineering to today’s workloads
Since last year’s public release of GenAI, people around the globe have been using it to research, tell stories, compose essays, create music and art, and write software code. ChatGPT, the popular chatbot from OpenAI, took only two months to reach 100 million monthly active users, making it the fastest growing consumer application in history.
But what’s behind GenAI and the AI infrastructure — the hardware, software, and firmware that makes it all run? The “cloud titans” running that infrastructure, –Google, Meta, Microsoft, and Amazon – make up more than 80 percent of the market. With the fast adoption of GenAI, these cloud titans are now “AI titans.”
That’s because to successfully support those using GenAI, these titans must support incredibly large internal workloads. They have vast amounts of data on which large language models, or LLMs, can be used to advance AI technologies in areas such as virtual assistants, chatbots and recommendation engines. In supporting commercial businesses, there is an expanding array of new areas in which LLMs can play a role, such as customer operations, supply-chain management, R&D and cybersecurity.
Because LLMs are trained on vast sets of data, AI infrastructure requires specialized compute accelerators that are interconnected to form massive clusters. In the infrastructure hardware world, most people know GPUs — general purpose graphics processing units. Then there are XPUs, “X” application-specific purpose units that are becoming critical for AI infrastructures. AI-specific XPUs are called custom AI accelerators because, literally, that’s what they are: custom chips that provide the processing power to drive AI infrastructure.
XPU Benefits and Customer Partnerships
Companies running AI infrastructure want a custom XPU when they have internal workloads critical for specific applications. By customizing the architecture of their accelerator, the (XPU to XPU) IO bandwidth and the memory bandwidth are able to process those application workloads at larger volumes, more efficiently than with general silicon.
The benefits of these custom chips can be explained by a simple equation: performance divided by total cost of ownership, or TCO. TCO is the sum of the cost of the chip, the power, and the infrastructure that puts it together.
At Broadcom, we work with our customers to customize their architecture so they can maximize the performance of their AI infrastructure and internal workloads. Optimizing hardware also leads to a smaller, less expensive, and reduced footprint. Customers that engage with us to co-develop these custom chips tend to save billions of dollars of capital expenses because they are designed to perform exactly as planned, with the right ratio of accelerator, memory bandwidth, and I/O bandwidth. Optimizing the hardware for AI infrastructure also means lowering the power and, therefore, the overall cost of these custom chips.
Taken together, these advantages combine to make Custom XPUs the ideal financial and engineering solution for consumer AI.
The Broadcom Engagement Model
Custom AI chips have essentially four layers: compute, memory, network I/O, and a reliable packaging technology. The complex architecture, along with intricacies in packaging, require us to look at what our customers want to accomplish with their AI workloads – three, five, and even 10 years into the future.
Working with our customers, we have developed, optimized, and automated a flow to build the accelerators as small, fast, and with as minimal space as possible. This is a shared responsibility owned by both Broadcom and the customer.
Broadcom owns the memory layer. We create the right memory solutions, the right connectivity, cooling, testing, and reliability qualification to ensure that the AI accelerators are reliable and ready to go. It’s worth noting that we also are running these interfaces significantly faster than any current standard.
We own the software tools that allow us to quickly put together chiplets to make the IOs wider or thinner to match the exact precision and ratio customers need for their AI workloads. This is the flexibility brought by software automation that we’ve also developed over the years.
Our accumulated knowledge over the last decade of co-developing XPUs uniquely positions us to quickly assist our customers. For example, we have finessed the mechanical, thermal, and electrical flows, and fine-tuned our testing and automation to streamline design and development.
Broadcom has the hardware, firmware, and software for a variety of networking solutions that can be included in the XPU as plug-and-play chiplets. This allows our customers to emulate, simulate, and take the accelerator to production faster and more efficiently.
Summing it up: Our Custom XPU engagement model allows our partners to get the best quality, performance, and TCO tuned for their workloads.
The Broadcom Advantage
Broadcom is committed to building on its decade-long industry leadership in custom silicon. We know what custom silicon builds require. That is why we have such a large and proactive investment in R&D – about $3 billion annually -- in creating technology of the future. The vast majority of this investment is prioritized for the AI market.
For example, our Co Package Optics Cores (proven in our latest XPU) can save 80 watts of system power per XPU. Imagine the benefits of putting a million of these accelerators together with that power and cost savings.
We continue to lead in areas such as high speed SERDES, chip to chip connectivity, high speed memory, 2.5 and 3D packaging, and time to market, at an annual cadence or faster, depending on customer needs.
Our objective is to continue to drive the best performance for AI workloads in XPUs at the lowest possible TCO for our customers. There are always challenges ahead, but we know that we have the expertise and experience to meet whatever comes next, to help our customers achieve even greater success.
To learn more and watch my presentation, please visit our page here.
Topics
xpu gpu clusters AI custom ai chips broadcom ai ai infrastructure