The energy use isn’t that extreme. A forward pass on a 7B can be achieved on a Mac book.
If it’s code and you RAG over some docs you could probably get away with a 4B tbh.
ML models use more energy than a simple model, however, not that much more.
The reason large companies are using so much energy is that they are using absolutely massive models to do everything so they can market a product. If individuals used the right model to solve the right problem (size, training, feed it with context etc. ) there would be no real issue.
It’s important we don’t conflate the excellent progress we’ve made with transformers over the last decade with an unregulated market, bad company practices and limited consumer Tech literacy.
The energy use isn’t that extreme. A forward pass on a 7B can be achieved on a Mac book.
If it’s code and you RAG over some docs you could probably get away with a 4B tbh.
ML models use more energy than a simple model, however, not that much more.
The reason large companies are using so much energy is that they are using absolutely massive models to do everything so they can market a product. If individuals used the right model to solve the right problem (size, training, feed it with context etc. ) there would be no real issue.
It’s important we don’t conflate the excellent progress we’ve made with transformers over the last decade with an unregulated market, bad company practices and limited consumer Tech literacy.
TL;DR: LLM != search engine