The carbon emissions from GPT-3 were 500 times more than those from a flight from New York to San Francisco.
The use of LLMs has given rise to a further concern: the effect of training these models on the environment. Unbelievably, training huge models results in hundreds of tonnes of carbon dioxide being released into the atmosphere. The carbon dioxide-equivalent emissions created by GPT-3 reached at 502 tonnes in 2022, the highest when compared to similar-parameter trained models, according to the sixth edition of AI Index Report 2023 issued by Stanford University.
The analysis hasn’t taken into account the most recent GPT-4 model, which might make things worse in this case. Notably, OpenAI has kept the size of its parameters a secret from the general population. Different metrics are used by researchers to estimate the carbon emissions produced by AI systems. This covers the quantity of parameters required to train the model, the efficiency of power utilisation in data centers, and the carbon intensity of the grid. The environmental impact, carbon emissions, or even parameter size were not mentioned by OpenAI in its most recent technical publication.
The GPT-3 model had the greatest emission when four LLM models were examined for the AI Index Report. It outperformed Gopher, an open-source model developed with substantial 280B parameters. With the same parameters as GPT-3, the multilingual language open model BLOOM produced 25 tonnes of carbon in 2022, which was 20 times less than GPT-3. The open pre-trained language model (OPT) from Meta used the least energy and generated only one-seventh as much carbon dioxide as GPT-3.
AI to Cut Down on Energy?
To tackle the high levels of energy consumption in AI systems, AI is currently being tested. While developing LLM models will require energy, experiments using reinforcement learning for managed commercial cooling systems have been made. Data centre energy efficiency is a goal of new reinforcement learning models like DeepMind’s BCOOLER (BVE-based Constrained Optimisation Learner with Ensemble Regularization).
Two actual energy-saving facilities have been the subject of live trials by DeepMind and Google. At the two experiment sites, the experiment revealed energy reductions of 9% and 13%.
Train with a weaker GPU
There are initiatives underway to reduce the large carbon footprints that LLM models produce. Studies on lowering the computation required to run these models have been taken into consideration. FlexGen, a high throughput generation engine for executing huge language models with constrained resources like a single commodity GPU, was recently released by AI research students. FlexGen searches for the most effective way to store and retrieve tensors using a linear programming optimizer. FlexGen can boost throughput by compressing these weights and enabling bigger batch sizes. FlexGen was able to run OPT-175B on a single 16GB GPU with a high throughput.
DistilBERT, a method for NLP Pre-Training that allows the training of any question-answering system or models using a single GPU, is a “distilled version” of BERT. DistilBERT is a more lightweight, quick, and affordable variation of BERT. It uses 40% less parameters and runs 60% faster while maintaining over 95% of BERT’s performances.
Because there are fewer training parameters required for smaller models, there may be fewer emissions as a result of this breakthrough. A foundation model by Meta AI called LLaMA, with parameters ranging from 7B to 65B, was released. Despite being ten times smaller than GPT-3, the LLaMA-13B is said to outperform it.
