• Dave.@aussie.zone
          link
          fedilink
          arrow-up
          0
          ·
          edit-2
          4 months ago

          I’m just waiting for the closed-source AI industry to have their Black Swan moment.

          Something’s going to come out of left field from the open AI community and all that investment in proprietary models and mega-compute is going to be rendered useless.

          • eleitl@lemmy.zip
            link
            fedilink
            arrow-up
            0
            ·
            4 months ago

            You still need the mega-compute. Even for inference, 1.5 terabytes of RAM in modern servers isn’t cheap.

            • Dave.@aussie.zone
              link
              fedilink
              arrow-up
              0
              ·
              4 months ago

              You’re not thinking black-swan enough.

              You’re typing your comments using a blob of goo with about a hundred million neurons in it that cycles under a hundred hertz and draws less than 20 watts.

              I don’t think that we’ll be running packs of goo in our PCs any time soon. But I do think some entirely different way of looking at the problem will emerge that will reduce computational requirements by many orders of magnitude. And it won’t involve gigantic statistical engines trying to find the best average response to a question.

              • eleitl@lemmy.zip
                link
                fedilink
                arrow-up
                1
                ·
                4 months ago

                There are 86 billion neurons in the human brain and 16 billions in the cerebral cortex. More importantly, there are 100-1000 trillion synapses, and we know that parts the dendritic tree do independent computations as well. The computations are not synchronized with a global clock, but 1 ms events correspond to a kHz refresh rate and we know spikes can do temporal coding. Meanwhile, the best we can do is Cerebras CS-3 and with WSI at 5 nm there isn’t much more where that came from.

                Try doing the math on how many CS-3 you’d need to represent and refresh the above.

      • nosuchanon@lemmy.world
        link
        fedilink
        arrow-up
        0
        ·
        4 months ago

        It’s not that complicated. Recreating training data via distillation is basically asking structured questions and recording the responses and reformatting that to use as cleaned “good” training data. Much less energy and compute intensive than creating the training data on your own.

        I think I remember reading somewhere how Chinese research’s do this by basically using bots and spreading out the distillation to many source queries.

        • ☆ Yσɠƚԋσʂ ☆@lemmy.mlOP
          link
          fedilink
          arrow-up
          1
          ·
          4 months ago

          The process takes time because even when you’re distilling answers, you still need to actually do reinforcement training on the model. And given that Fable and GPT 5.6 just came out there simply hasn’t been much time to do that. However, models like Kimi also do better than Fable or GPT on a lot of tasks, which means it’s not just distillation but also difference in architecture. You can watch this talk from Kimi founder to see how Kimi was actually trained and why it performs well.

          It’s also absolutely hilarious that people think only Chinese companies use distillation, as if Anthropic or OpenAI are above that or something. Not to mention that they basically ignored copyrights on all the data the siphoned and are now crying that people aren’t respecting their terms of use.