OpenAI Trumps Nvidia

A hardware team starts designing next generation chips around 3 years in advance. They however do not know the exact customer expectations from this chip. Customers are hesitant to share their future software models, because they (a) haven’t yet designed them or (b) don’t want to share them due to IP considerations.

The hardware team thus has to estimate specs of the next generation of chips by using inputs from their internal customer-facing and developer-relationship teams to provide them the current customers’ use-cases.

Whoever has customers at the cutting-edge of software today has the best knowledge of the software that is working. In other words, whoever has product-market fit in the hardware space can best approximate the needs of the software customers in the future.

Similarly, whoever has product-market fit in the consumer AI space can best predict the software models of the future, and thus also best predict the specs of the next generation hardware models.

If this logic holds true and based on first principles, Nvidia - which is at the cutting edge of the hardware - can potentially be displaced by OpenAI - if it chooses to vertically integrate. OpenAI will have enough scale on the inference that it can afford investing into building its own inference chip for a model that they will develop 3 years down the line. Since an OpenAI chip can be coupled tighter to the model architecture - in the same way Google’s TPUs are tailored for their ML workloads - inference costs can be reduced further. Nvidia can’t match those optimizations because they have to serve a wider audience.

OpenAI will have to acquire talent, which is not trivial, but certainly possible. They have already done something similar by hiring software talent away from traditional tech companies. They should be able to do the same for hardware companies as well. Otherwise, the industry will continue to experience higher inference costs due to the asymmetric information between model developers and chip designers.

While Nvidia continues to innovate in this duration, OpenAI can withhold information of its upcoming models. With the advent of RL, While there is substantial learning that Nvidia has obtained over the years, this moat will gradually reduce with AI and RL.