1. Are We Asking AI Models the Right Questions?
While artificial intelligence (AI) language models can improve in many ways, a recent paper[i] by researchers from Microsoft and OpenAI illustrates the importance of posing questions to models in the right way. As shown below in the gray dotted line on the left, earlier this year smaller AI models “fine-tuned” on specialized data caught up to OpenAI's state-of-the-art GPT-4 model on a medical licensing exam test. As shown by the blue line on the left and on the right chart, respectively, systematic questions not only restored GPT-4's lead but also improved its performance across a variety of medical and biological problems—not just the questions posed by medical licensing board exams.
Source: Nori et al. 2023. (See endnote i.) For informational purposes only and should not be considered investment advice or a recommendation to buy, sell, or hold any particular security or cryptocurrency. Past performance is not indicative of future results.
In our view, this research demonstrates the power of “prompt engineering,” the practice of modifying model queries to improve answers. While seemingly a function of more precise language in the questions, the source of the improvement is more complex. When feeding math questions to Google’s Palm 2 model, for example, researchers found that the model answered them correctly only 30% of the time but, by asking the same model to "take it step by step," it answered the same questions correctly 60% of the time. Asked to "take a deep breath and take it step by step" with the same questions, the model became a “B” student. Perhaps the reason for the improved performance is in the source material directing people to “take a deep breath,” which would include instructions to use step-by-step logic to reach answers. While we are not sure, this “chain of thought” prompting technique does seem to improve model performance universally, as shown below.
.png)
Source: ARK Investment Management LLC, 2023, based on data from Yang et al.[ii] For informational purposes only and should not be considered investment advice or a recommendation to buy, sell, or hold any particular security or cryptocurrency. Past performance is not indicative of future results.
The researchers systematically harnessed a number of these techniques to improve GPT-4’s performance in a range of subjects, often raising it to expert levels with little incremental cost. On an error reduction basis, they improved GPT-4's performance on the medical licensing exam almost as much as GPT-4 improved upon that of GPT-3.5. Important in that case, however, training was ~10 times more expensive for GPT-4 than GPT-3.5 while, in the medical examples discussed above, the increased number of queries required little more than a few more tokens, as shown below.
Source: ARK Investment Management LLC, 2023, based on data from Yang et al. 2023. For informational purposes only and should not be considered investment advice or a recommendation to buy, sell, or hold any particular security or cryptocurrency. Past performance is not indicative of future results.
These practical techniques suggest that researchers have significant room to optimize models and maximize performance across the AI space. We look forward to learning more about the research in this space.
3. Tesla Has Delivered Its First Cybertrucks
Last Thursday, as it delivered the first Cybertrucks to customers eager to learn about the price and range for each trim, Tesla disclosed that an All-Wheel Drive Cybertruck with a 340-mile range costs $79,990—pre-tax credits—but, for an additional ~$16,000, will extend the range to 470-miles. Cybertruck’s pricing is competitive with industry-wide average transaction prices—$66,184 for full-size trucks and $77,906 for full-size SUVs as of October 2023.[viii]
In our view, the Cybertruck is helping Tesla to achieve two of its goals simultaneously. First, it is a potentially profitable research and development platform, showcasing technology innovations like “steer-by-wire" and 48-volt electrical architecture, which could appear in other future vehicles. Second, as the crowds at Tesla showrooms made abundantly clear,ix the Cybertruck is a marketing machine, its innovations defining a new vehicle category and amplifying Tesla’s technological lead over competitors.
[i] Nori et al. 2023. “Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine.” arXiv.
[ii] Yang, C et al. 2023. “Large Language Models as Optimizers.” arXiv.
[iii] U.S. Food & Drug Administration. 2023. “FDA Investigating Serious Risk of T-cell Malignancy Following BCMA-Directed or CD19-Directed Autologous Chimeric Antigen Receptor (CAR) T cell Immunotherapies.”
[iv] According to Moffitt Cancer Center, “Autologous CAR T-cell therapy uses a patient’s own immune cells, while allogeneic T-cell therapy uses T cells from donor blood or umbilical cord blood.” See Moffitt Medical Minute. 2022. “Autologous vs. Allogeneic CAR T.”
[v] Boettner, B. 2023. “Fine-tuning stimulation doses to deficiencies in patient-specific CAR-T cells, using artificial antigen-presenting scaffolds, enables manufacturing of more potent CAR-T cell products.” Wyss Institute.
[vi] Bristol Myers Squibb. 2023. “Blood Cancers.”
[vii] Frey, N. V. 2021. “Relapsed ALL: CAR T vs transplant vs novel therapies.”
[viii] Cox Automotive. 2023. “Data Tables for October 2023 Kelley Blue Book Average Transaction Prices Report.”
[ix] Lambert, F. 2023. “Tesla Cybertrucks in Showrooms Attract Large Crowds.” Electrek.