The interest in large language models (LLMs) has surged by 1,310%, according to a new study by Databricks, the data and AI company.
The 2023 State of Data + AI report analyses anonymised usage data from more than 9,000 global Databricks customers to provide a comprehensive look at organisations’ data and AI initiatives.
“The historic surge of interest in large language models (LLMs) since ChatGPT launched to the public late last year has made the topic inescapable. Not only is the technology improving at an unparalleled cadence, but companies are also building their own models like never before.” says Nick Eayrs, Vice President, Field Engineering, at Databricks Asia Pacific & Japan. “Now, predictive models are underpinning mission-critical tasks, giving organisations significant competitive advantage and allowing them to provide highly differentiated products and services.”
The latest report by Databricks uncovers exactly where enterprises find themselves in this transformation, as well as the platforms and tools they’re using to take advantage of it.
- The hype around LLMs is real: From the end of November 2022 to the beginning of May 2023, the usage of SaaS LLMs, which are used to access models like OpenAI, grew exponentially with Lakehouse customers at a rapid 1310%. Transformer-related libraries like HuggingFace (an NLP toolkit and model hub), which are used to train homegrown LLMs and were in demand even before the launch of ChatGPT, grew 82% within the same time frame.
- Data transformation and integration is more vital than ever: The fastest growing tools on Databricks are dbt (206%) and FiveTran (181%). But of the 10 most popular data and AI products, six are data integration tools, including Informatica and Qlik, making it the fastest growing market on the Databricks Lakehouse.
- Companies eye open source: When looking at the most popular data and AI products, Microsoft Power BI and Plotly reign above the rest. But organisations are showing a strong pull to open technologies; 8 of the 10 most popular data and AI products are based on open source software, including dbt, Huggingface and GeoPandas.
- Enterprises are doing more AI projects than ever before – and getting better at it: The amount of models in production, or those that are actually used in operations, grew 411% year-over-year, while the number of experimental projects grew 54%. The data also shows that, on average, one in three experimental models are a candidate for the real-world, compared to one in five last year, suggesting organisations are getting better at building and scaling these projects.
- AI is growing, but don’t forget traditional data analytics: Last year, Power BI was the most popular program running on top of the Lakehouse. Meanwhile, serverless data warehousing with Databricks SQL grew 146% last year.
While it’s still early days, these emerging trends are bound to define the future of AI. And business leaders need to pay attention. It’s never been more clear: the companies that harness the power of DS/ML will lead the next generation of data.
You can read the full report here.
