Running llama-2-7b-chat at 8 bit quantization, and completions are essentially at GPT-3.5 levels on a single 4090 using 15gb VRAM. I don’t think most people realize just how small and efficient these models are going to become.

[cut out many, many paragraphs of LLM-generated output which prove… something?]

my chatbot is so small and efficient it only fully utilizes one $2000 graphics card per user! that’s only 450W for as long as it takes the thing to generate whatever bullshit it’s outputting, drawn by a graphics card that’s priced so high not even gamers are buying them!

you’d think my industry would have learned anything at all from being tricked into running loud, hot, incredibly power-hungry crypto mining rigs under their desks for no profit at all, but nah

not a single thought spared for how this can’t possibly be any more cost-effective for OpenAI either; just the assumption that their APIs will somehow always be cheaper than the hardware and energy required to run the model

  • maol@awful.systemsEnglish
    1·
    1 year ago

    Who decided that this point on the climate change graph was a good point at which to spend millions of dollars on AI.

  • bobtreehugger@awful.systemsEnglish
    1·
    1 year ago

    And this isn’t even the expensive part – training, this is just inference.

    Can’t wait for this fad to be over

      • self@awful.systemsOPEnglish
        1·
        1 year ago

        why should this be bad tech ?

        because it is godawful at:

        • replace lawyering jobs
        • save on grafic design costs
        • no more language teachers
        • youtube videos can be transribed in text format and used as learning material
        • froztbyte@awful.systems
          1·
          1 year ago

          one of the things that I like using as an example here is: just make it do something it isn’t currently trained on

          e.g. try to make it render content in zulu or isixhosa or [insert list of thousands of things that the developers barely/never touch] - it’s near guaranteed to have been trained on a very, very narrow set of that subject (if anything at all)

          “then just train it on more data” comes the refrain

          you: “okay, find me sufficient data of that”

          them: “it’s just a curation problem”

          you: “then who will create that?”

          the absolute very minimum of thinking beyond the second order just so entirely evades so many of these promptfans it’s astounding

          edit: TIL lemmy doesn’t do single newlines well

          • self@awful.systemsOPEnglish
            2·
            1 year ago

            I keep flashing back to eliezer being smug on Twitter about how good ChatGPT is at chess, and it turns out once you get past book openings and extremely well-documented games, it completely shits the bed and stops acting like it knows the rules of chess or even basic chess notation. and this is a very obvious outcome if you know how LLMs work, but most promptfans don’t

    • 200fifty@awful.systemsEnglish
      1·
      1 year ago

      for real though, i keep saying important people specifically say “AI will help with climate change” and like… how, dude? by burning a ton of energy to think really hard about it with its magic brain powers? like, what is the actual concrete help here supposed to be, for real. is this just the new “crypto incentivizes switching to green energy”? :/

  • al177@lemmy.sdf.orgEnglish
    0·
    1 year ago

    It’s a shame that analog inference accelerators are taking so long to hit the market. GPUs are way too expensive and power hungry for inference when you don’t need the ability to train a network.

    • self@awful.systemsOPEnglish
      1·
      1 year ago

      oh totally, upgrading from GPUs to ASICs will really increase my hash rate mining profits number of concurrent conversations