2. Google Made Some Unforced Errors During Its Highly Anticipated Gemini Launch
Last week, Google introduced Gemini,[vi] its latest suite of foundation models. Built from the ground up to be multi-modal—including images, audio, video, and text—Gemini purportedly is the first AI model to score higher than expert humans on the Massive Multitask Language Understanding (MMLU) benchmark, thanks to its better inherent reasoning and generation capabilities across various media.
Here’s the catch! To capture that headline, Google used different prompting techniques in comparing Gemini Ultra to GPT-4. Only when prompted by a special technique called “Chain-of-Thought@32 (Uncertainty-Routed),” which Google developed, did Gemini Ultra outperform both GPT-4 and expert humans. Based on the standard “5-shot” MMLU benchmark method described in Open-AI's paper, GPT-4 outperformed Gemini Ultra. In Google’s technical report,[vii] this important distinction became clear, as shown below.
Source: Google 2023.[viii] For informational purposes only and should not be considered investment advice or a recommendation to buy, sell, or hold any particular security or cryptocurrency. Past performance is not indicative of future results.
In our view, Google compared apples to oranges in declaring Gemini Ultra the superior model. Moreover, a video[ix] showcasing Gemini’s vision and reasoning capabilities had been edited[x] significantly: it did not include full prompts, nor did it delineate between the contribution to results of the model and the video editor.
Perhaps caused by Google’s marketing team, these blunders risk undercutting the credibility of DeepMind, Google’s formidable AI team. The engineers at DeepMind have contributed to many significant technical breakthroughs and “ah-ha” moments in AI over the years and seemingly would not want to blemish their record with this kind of deception. Perhaps Google has ceded too much power to marketing in response to competitive pressure from OpenAI in the monumental race toward AGI (Artificial General Intelligence)!
3. AMD Is Competing With Nvidia for Inference Workloads
Last week, AMD officially launched[xi] its MI300X accelerator which, according to the company, outperforms Nvidia's H100 by a factor of 1.4x in inferencing Llama 2 70B. While that trails NVIDIA's next generation H200's 1.9x improvement over the H100, AMD is likely to price the MI300X significantly lower than both the H100 and H200. If AMD were to price the MI300X at $18-$19,000,[xii] and if H100s and H200s were to sell in the $25-$40,000[xiii] range, then not only would the MI300X offer better performance per dollar of capex than both the H100 and H200, as shown below, but it also could be accretive to AMD’s margins.
Source: ARK Investment Management LLC, 2023, based on data from Nvidia as of December 8, 2023. Forecasts are inherently limited and cannot be relied upon. For informational purposes only and should not be considered investment advice or a recommendation to buy, sell, or hold any particular security or cryptocurrency. Past performance is not indicative of future results.
AMD also touted the performance improvement in its open-source software ecosystem with support from Meta, OpenAI, and others. While AMD’s ROCm software is gaining traction, Nvidia’s software moat remains formidable. Highlighted[xiv] during Nvidia’s most recent earnings call, the TensorRT-LLM SDK (Software Development Kit) promises to boost LLM inferencing performance 2x on Nvidia hardware. Software can improve performance per dollar of capex on AI inferencing hardware significantly, as shown on the right side of the chart above. In our view, software will remain an important battleground for GPU providers as the applications built around LLMs drive demand for efficient compute.
4. “Animate Anyone” Could Disrupt Video Production
Two weeks ago, the Institute for Intelligent Computing and Alibaba Group published an image-to-video model called Animate Anyone.[xv] The model transforms an input reference image and a pose sequence into a video. Unlike in previous image-to-video research, Animate Anyone introduced a feature extraction network called ReferenceNet that captures details from the original reference image in each video frame generated.
Trained on 5,000-character video clips, 340 dancing TikTok videos, and 500 fashion videos, and preserving image detail across multiple frames, Animate Anyone seems to outperform existing image-to-video generation methods in creating detailed, consistent, and continuous short form videos across a few genres. In our view, as the supply of user-generated content continues to boom, the algorithmic curation of entertainment will silo users even more narrowly into their own content microcosms, increasingly fragmenting viewership. Interestingly, The Verge reported[xvi] that its editor had not seen any of TikTok’s top 10 videos for the year 2023.
As a result, shared entertainment—either online or in-person—could become even more valuable over time. Perhaps to the dismay of consumers, live sports, live music, and social gaming could continue to benefit from higher priced tickets and advertising.
[i] U.S. Food & Drug Administration. 2023. “FDA Approves First Gene Therapies to Treat Patients with Sickle Cell Disease.
[ii] Centers for Disease Control and Addiction. 2023. “Data & Statistics on Sickle Cell Disease.”
[iii] National Heart, Lung, and Blood Institute. 2023. “What Is Sickle Cell Disease?” [iv] Urman, A. 2023. “The FDA AdComm Looks Promising For Exa-Cel.” ARK Disrupt. ARK Investment Management LLC.
[v] CRISPR Therapeutics. 2023. “CRISPR Therapeutics Announces Completion of FDA Advisory Committee Meeting for Exagamglogene Autotemcel (exa-cel) for Severe Sickle Cell Disease.”
[vi] Google Deep Mind. 2023. “Welcome to the Gemini Era.”
[vii] Google. 2023. “Gemini: A Family of Highly Capable Multimodal Models.”
[viii] Ibid.
[ix] Google. 2023. ”Hands-on with Gemini: Interacting with multimodal AI.” YouTube.
[x] Jones, A. 2023. “Fun fact: that viral-ish Gemini demo video is basically entirely fake.” X.
[xi] AMD. 2023. “Advancing AI.”
[xii] Patel, D. and Kostovic, A. 2023. “AMD AI Software Solved — MI300X Pricing, Performance, PyTorch 20.0, Flash Attention, OpenAI Triton.” Semianalysis. [xiii] Schreiner, M. 2023. “Nvidia's H100 GPU sells like hot cakes with high profit margins.” The Decoder.
[xiv] SA Transcripts. 2024. “NVDIA Corporation (NVDA) Q3 2024 Earnings Call Transcript.” Seeking Alpha.
[xv] Hu, L. et al. 2023. “Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation.”
[xvi] Sato, M. 2023. “TikTok’s biggest hits are videos you’ve probably never seen.” The Verge.