Episode 30 As AI Ubiquity Looms, CAST AI Balances Cloud Efficiency & Environmental Impact

Tech Uncensored - An Altitude Accelerator Podcast

Today, we are witnessing the price of progress. As generative AI swiftly evolves amidst a booming landscape of adoption, the marvels of artificial intelligence are met with astounding costs and challenges. The allure from the VC community and tech giants, who have invested billions of dollars into startups specializing in generative AI technologies have not considered the underlying reality of these high costs that threaten this current boom.

As of June 2023, ChatGPT has received 60 million visits daily, with 10 million queries per day. As of April 2023, it was estimated that to run ChatGPT would cost $70,000 per day at an average cost of $0.36 per question. In June, however, “Tom Goldstein, an AI ML Professor at Maryland University, has estimated the daily cost of running ChatGPT to be approximately $100,000 and the monthly cost to be USD$3 million.”

This recent article profiled one startup, Latitude, which found itself grappling with exorbitant bills as their AI-powered games, like AI Dungeon, gained popularity. Latitude's text-based role-playing game utilized OpenAI's GPT language technology, resulting in soaring costs proportional to the game's usage. Content marketers' unexpected usage of AI Dungeon for generating promotional copy further exacerbated the startup’s financial strain.

One of the primary reasons for the high cost of generative AI is the substantial computing power required for “training and inference.”

I met with Laurent Gil, former lead of Oracle’s Internet Intelligence Group and current Cofounder of CAST AI, which is an ML powered cloud optimization platform that analyzes millions of data points, looking for the optimal balance of high performance at the lowest cost. CAST AI determines how much you can save, then reallocates your cloud resources in real time to hit the target with no impact to performance.