• Kuvwert@lemm.ee
          link
          fedilink
          English
          arrow-up
          3
          ·
          edit-2
          2 days ago

          https://ibb.co/wVNsn5H

          https://ibb.co/HpK5G5Pp

          https://ibb.co/sp1wGMFb

          https://ibb.co/4wyKhkRH

          https://ibb.co/WpBTZPRm

          https://ibb.co/0yP73j6G

          Note that my tests were via groq and the r1 70B distilled llama variant (the 2nd smartest version afaik)

          Edit 1:

          Incidentally… I propositioned a coworker to answer the same question. This is the summarized conversation I had:

          Me: “Hey Billy, can you answer a question? in under 3 seconds answer my following question”

          Billy: “sure”

          Me: “How many As are in abracadabra 3.2.1”

          Billy: “4” (answered in less than 3 seconds)

          Me: “nope”

          I’m gonna poll the office and see how many people get it right with the same opportunity the ai had.

          Edit 2: The second coworker said “6” in about 5 seconds

          Edit 3: Third coworker said 4, in 3 seconds

          Edit 4: I asked two more people and one of them got it right… But I’m 60% sure she heard me asking the previous employee, but if she didnt we’re at 1/5

          In probably done with this game for the day.

          I’m pretty flabbergasted with the results of my very unscientific experiment, but now I can say (with a mountain of anecdotal juice) that with letter counting, R1 70b is wildly faster and more accurate than humans .

            • rdri@lemmy.world
              link
              fedilink
              English
              arrow-up
              5
              ·
              edit-2
              3 days ago

              It didn’t, I just wanted a short reply. Though it failed when I asked again in the same chat. But when asked to split the word to 2 parts it became sure that the correct answer is 3.

              Here is the fun part

              Certainly! Let’s split “strawberry” into two equal parts and count the 'r’s in each:

              1. First half: “strawb”
                Letters: S, T, R, A, W, B
                'r’s: 1 (at position 3)

              2. Second half: “erry”
                Letters: E, R, R, Y
                'r’s: 2 (at positions 2 and 3)

              Now, summing them: 1 (first half) + 2 (second half) = 3 'r’s.

              Wait, this contradicts my earlier correction. It seems I made another mistake. Let’s carefully recount:

              • First half: “strawb”
                'r’s: 1 (at position 3)

              • Second half: “erry”
                'r’s: 1 (only one ‘r’ at position 2; the third letter is ‘r’ but already counted in the first half).

              Wait, no—this is getting confusing. Let’s re-examine the entire word:

              Full word: S, T, R, A, W, B, E, R, R, Y
              'r’s: Positions 3, 8, and 9 → 3 'r’s.

              I apologize for the back-and-forth. The correct total is 3 'r’s in “strawberry.” Thank you for your patience as I worked through this! 😊

              • blakenong@lemmings.world
                link
                fedilink
                English
                arrow-up
                5
                ·
                3 days ago

                That’s a lot of processing just to count letters. Hopefully it can add numbers without splitting the number ;)

                • ikt@aussie.zone
                  link
                  fedilink
                  English
                  arrow-up
                  2
                  ·
                  2 days ago

                  That’s a lot of processing just to count letters

                  feel free to ask Google/Bing/Your favourite search engine to do the same :P

                • Pieisawesome@lemmy.world
                  link
                  fedilink
                  English
                  arrow-up
                  4
                  ·
                  2 days ago

                  It’s because LLMs don’t work with letters. They work with tokens that are converted to vectors.

                  They literally don’t see the word “strawberry” in order to count the letters.

                  Splitting the letter probably separates them into individual tokens

      • blakenong@lemmings.world
        link
        fedilink
        English
        arrow-up
        9
        ·
        edit-2
        3 days ago

        No. It literally cannot count the number of R letters in strawberry. It says 2, there are 3. ChatGPT had this problem, but it seems it is fixed. However if you say “are you sure?” It says 2 again.

        Ask ChatGPT to make an image of a cat without a tail. Impossible. Odd, I know, but one of those weird AI issues

        • SoftestSapphic@lemmy.world
          link
          fedilink
          English
          arrow-up
          7
          ·
          3 days ago

          Because there aren’t enough pictures of tail-less cats out there to train on.

          It’s literally impossible for it to give you a cat with no tail because it can’t find enough to copy and ends up regurgitating cats with tails.

          Same for a glass of water spilling over, it can’t show you an overfilled glass of water because there aren’t enough pictures available for it to copy.

          This is why telling a chatbot to generate a picture for you will never be a real replacement for an artist who can draw what you ask them to.

      • Kuvwert@lemm.ee
        link
        fedilink
        English
        arrow-up
        5
        ·
        3 days ago

        Non thinking prediction models can’t count the r’s in strawberry due to the nature of tokenization.

        However openai o1 and deep seek r1 can both reliably do it correctly