While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

  • LuckyLu@lemmy.world
    link
    fedilink
    arrow-up
    320
    ·
    1 year ago

    Very impressed by how quickly action has been taken by this and other instances to patch the issue.

    • Darkard@lemmy.world
      link
      fedilink
      arrow-up
      19
      ·
      edit-2
      1 year ago

      Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.

      I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.

      In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)

      • LuckyLu@lemmy.world
        link
        fedilink
        arrow-up
        15
        ·
        1 year ago

        I’m on iOS with the Memmy app. It’s a work in progress that’s officially unfinished so I’m not surprised but it has also been a bit buggy. Doesn’t seem that I can log out without deleting and reinstalling the app so hopefully this doesn’t happen too often XD

        • Carnelian@lemmy.world
          link
          fedilink
          arrow-up
          11
          ·
          1 year ago

          So I was actually just struggling with that myself, also in the Memmy app in case that isn’t clear

          What I did was add my account (again)

          There was no warning or anything, and it populated the list with two of me.

          At that point, a “delete account” option appeared under both of them. So I guess in normal circumstances, it wants you to keep one account around at all times?

          I deleted one of them, and the app basically reinitialized. Both were gone and it showed me the welcome screen.

          I logged back in, and now everything is back to normal

          • NaN@lemmy.blahaj.zone
            link
            fedilink
            arrow-up
            3
            ·
            1 year ago

            I just did edit account and then saved, it seemed to trick it into logging in again (secrets on my instance were also reset).

          • scarabic@lemmy.world
            link
            fedilink
            arrow-up
            2
            ·
            1 year ago

            Ah interesting. I’ve had multiple accounts from the start so it was much easier for me. Just removed my main account and added it back.

          • Pandantic@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            edit-2
            1 year ago

            I did this, but I just didn’t delete either accounts and it worked fine. Idk if it’s detrimental to have two of the same but it worked for me.

          • LuckyLu@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            1 year ago

            Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.

          • JJROKCZ@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            1 year ago

            Finally I found good instructions, was about to delete and reinstall until I followed this!

          • LuckyLu@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            1 year ago

            Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.

        • Infernal_pizza@lemmy.world
          link
          fedilink
          arrow-up
          7
          ·
          1 year ago

          I found I didn’t actually have to log out, just go into account settings and reconfirm everything without changing it

        • scarabic@lemmy.world
          link
          fedilink
          arrow-up
          2
          ·
          1 year ago

          No you can. You just remove the account from the accounts list. It’s labeled “delete this account” which is scary but it just removes it from Memmy. You can add it right back and that logs you back in. Not a great experience.

          I sure hope this doesn’t happen a lot. This kind of barrier hurts site growth. I’ve managed a lot of large sites and seen a lot of bugs and when everyone gets logged out there is a measurable impact, and some folks never return. Just look at all the comments here saying “thank I didn’t know to do that.” For every one of those there are 100 people going “huh… Lemmy is down… oh well… on to something else…”

        • calaei@lemmy.world
          link
          fedilink
          arrow-up
          2
          ·
          1 year ago

          Go into account settings, clear your password, re-enter your password, save, go to feed and pull to refresh. That’s what worked for me.

        • No1ButtMe@lemmy.world
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          I was I able to upvote anything or subscribe. Seems like uninstalling and reinstalling fixed my issue

        • LazaroFilm@lemmy.world
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          For Memmy, I went to the accounts page in the settings. Click d on my lemmy.world account then to the page where you can change the password then navigated away. That reactivated the account. Maybe we should add a ticket on Memmy’s GitHub about reactivating cookies when there’s an issue. Or at least place à poput to double check credentials or something.

      • cookie@lemmy.world
        link
        fedilink
        arrow-up
        2
        ·
        1 year ago

        Oh, I was wondering why it was showing me as logged in but wouldn’t let me upvote due to not being logged in. Your liftoff psa just cleared that right up for me, thanks!

      • deweydecibel@lemmy.world
        link
        fedilink
        arrow-up
        2
        ·
        1 year ago

        In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)

        Good PSA. It took me a bit to figure it out, the app doesn’t make this obvious.

      • LuckyLu@lemmy.world
        link
        fedilink
        arrow-up
        2
        ·
        1 year ago

        Negative one upvotes would mean that enough people disliked me/another poster to bring my upvote total to zero. (Upvotes and likes are effectively the same thing, it’s just a naming convention). Reddit totals them up and seemingly Lemmy does as well.

        • grissee@lemm.ee
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          huh that’s weird (yes I meant negative one downvote), I already know that the total can be either positive or negative, but shouldn’t the upvote number and downvote number be either positive or zero? (for now I’ll just accept it as a lemmy bug/ inconsistencies between instances)

          • LuckyLu@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            1 year ago

            Nope, just like Reddit it’s a value that ranges between negatives and positives. If I get two thousand upvotes, positive 2k. If I get two thousand downvotes, negative 1999 (because iirc you start with one by default).

            Not exactly sure I understood what you meant by “either positive or zero”.

            • grissee@lemm.ee
              link
              fedilink
              arrow-up
              2
              ·
              1 year ago

              see your comment rn, it has 1 upvote (from yourself by default) and 0 dislike (so it’s not shown)

              but in the screenshot I sent above you got 287 upvote and minus -1 downvote (making your total 288) which is mathematically correct but seems like an unintended behavior

              for example this comment of mine normally have 9 upvote and 2 downvote (which is shown as a positive integer 2, not negative), making my total upvote 7

              • LuckyLu@lemmy.world
                link
                fedilink
                arrow-up
                2
                ·
                1 year ago

                Just occurred to me that the app I use also shows separate counters. I fooled myself into thinking it was a single counter.

                That’s interesting. Remember it’s a very new platform, minor bugs aren’t out of the ordinary.

  • ThisIsMyLemmyLogin@lemmy.world
    link
    fedilink
    arrow-up
    276
    arrow-down
    1
    ·
    1 year ago

    I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.

  • Ahmed@lemmy.world
    link
    fedilink
    arrow-up
    135
    ·
    1 year ago

    Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.

  • Vamp@lemmy.world
    link
    fedilink
    arrow-up
    101
    arrow-down
    2
    ·
    1 year ago

    what steps are being taken to ensure it doesn’t happen again? was any personal data compromised for users?

      • Vamp@lemmy.world
        link
        fedilink
        arrow-up
        50
        ·
        1 year ago

        Also I am curious, what’s the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?

      • BustedPancake@lemmy.world
        link
        fedilink
        arrow-up
        20
        ·
        1 year ago

        So all our cookies are negated now with the JWT changed, and we just needed to login again? Can attackers have stolen our cookies in order to use our accounts to post as if it was us? I’m sure they were only interested in admin cookies, so most others were “useless” to them? I see nothing wrong with my posts so I should be safe, right?

        • cantevencode@lemmy.world
          link
          fedilink
          arrow-up
          26
          ·
          1 year ago

          Prior to the JWT secret being rotated, yes, they could have authenticated as you. The tokens are now all invalid and useless

        • Rooki@lemmy.world
          link
          fedilink
          English
          arrow-up
          6
          arrow-down
          1
          ·
          1 year ago

          If you think they could change your password:

          YES, they could.

          They could have changed the email => “Forgot PW” and with that you lost ur account.

          • Xero@lemmy.world
            link
            fedilink
            English
            arrow-up
            5
            arrow-down
            3
            ·
            edit-2
            1 year ago

            I think I’ve lost my account, I clicked Forgot Password and nothing came into my mailbox. This account is the one I made just now.

            My old account:

            If you see that account post or comment on anything, please report it

            Edit: Nvm, I use another email to sign up for Lemmy and forgot about it

              • Xero@lemmy.world
                link
                fedilink
                English
                arrow-up
                5
                ·
                1 year ago

                actually nevermind, I forgot that I use a different email for Lemmy, I can log back in now

                • TheSmartDude@lemmy.world
                  link
                  fedilink
                  arrow-up
                  3
                  ·
                  edit-2
                  1 year ago

                  It happens to all of us. Additionally, assuming that you’ve come here recently, there’s not much data on it, and it being deleted will not be that much of a big deal.

          • linearchaos@lemmy.world
            link
            fedilink
            arrow-up
            17
            ·
            1 year ago

            I’m ok with the dicks but the threads are TOO FAR!!! shuffles off to the angry done**

            Thank you all for staying on top of it.

          • Milan@discuss.tchncs.de
            link
            fedilink
            arrow-up
            4
            ·
            1 year ago

            right after the update we also had most of the serverlist cleared except threads.net (which was the last one added so i assumed it was some bug) – otherwise nothing appears to be touched on this instance tho.

  • wazoobonkerbrain@lemmy.world
    link
    fedilink
    arrow-up
    96
    ·
    1 year ago

    IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.

  • trouser_mouse@lemmy.world
    link
    fedilink
    arrow-up
    92
    arrow-down
    2
    ·
    edit-2
    1 year ago

    First - really good summary and sounds like everyone is working hard.

    Cross posting the below comment.

    Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.

    There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I don’t really have enough familiarity with the regulation to discuss that one.

    If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.

    Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.

    For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.

    It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.

    • B16_BR0TH3R@lemmy.world
      link
      fedilink
      arrow-up
      23
      ·
      1 year ago

      If a valid browser token gets stolen like in this case, then MFA won’t do much good because the stolen token will already have been authenticated. Linus Tech Tips experienced the same thing recently, you can check out their channel.

      • trouser_mouse@lemmy.world
        link
        fedilink
        arrow-up
        7
        ·
        1 year ago

        That makes sense, thanks so much - there’s a few good explanations here which really help! Would it be right in saying that all affected servers should be logging off all users - some have but not sure if all.

        • Riskable@programming.dev
          link
          fedilink
          arrow-up
          7
          ·
          edit-2
          1 year ago

          The fix is to force the use of a new JWT encryption key which–when set–would immediately invalidate all existing user cookies, forcing all users to relogin.

          Lemmy has a few weaknesses related to their use of JWT in cookies that need to be addressed… The biggest one being that they use the same secret key for all user cookies (every user should have their own unique session key). I’m pretty sure that if they implemented that the scope of this vulnerability would be drastically reduced (but I haven’t looked at the precise mechanism of the vulnerability yet).

          They also need to provide tools in the GUI for admins and users to invalidate all issued sessions (cookies) and a mechanism for regularly rotating session secrets (the cookie currently lasts for a year and even if the session token gets regenerated it’ll still use the same secret).

          They also need to make the expiration times configurable so that security-focused servers can set short expiration times. Related, they need to force the use of unique secrets for every session (even if it’s the same user using different devices/apps).

        • B16_BR0TH3R@lemmy.world
          link
          fedilink
          arrow-up
          6
          ·
          1 year ago

          I guess that would depend on the specific case. If you physically went on my computer to steal my token or infected my computer with a virus to do it then we can assume that no other tokens have been compromised. But if the malicious actor has managed to steal tokens from the actual server (which seems to be the case here) and not the client then yes, as the admin I would certainly require that everyone log in again as a safety measure.

    • hawkwind@lemmy.management
      link
      fedilink
      arrow-up
      19
      ·
      1 year ago

      Out of curiosity, where would the regulators go for a case like this? There’s no “company” running it per. se.

    • VeganPizza69 Ⓥ@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      ·
      edit-2
      1 year ago

      Can 2FA be enabled for all users? I don’t see the link to activate it after saving.

      edit

      Yeah, this doesn’t work at all. The apps don’t open links anymore. I tried some github site that reads the link and generates a QR, but the codes don’t work. This is a complete waste of time.

      • nelrico@lemmy.world
        link
        fedilink
        English
        arrow-up
        6
        ·
        edit-2
        1 year ago

        Just reload the settings page after saving and you’ll see the activation link. Just now enabled 2FA for my account.

        • u/lukmly013 💾 (lemmy.sdf.org)@lemmy.sdf.org
          link
          fedilink
          English
          arrow-up
          5
          ·
          1 year ago

          Don’t log out! Open private tab and try logging in to test that it works. Lemmy uses SHA-256 TOTP digest which may not work correctly with some authenticators, only generating useless codes.

          • Muddybulldog@mylemmy.win
            link
            fedilink
            arrow-up
            2
            ·
            1 year ago

            The interface for TOTP need to be greatly improved as well. I made sure that I had two browsers logged in when I did it because the flow is so hinky. Not having a confirmation process was a bit nerve racking.

          • VeganPizza69 Ⓥ@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 year ago

            Yeah, this doesn’t work at all. The apps don’t open links anymore. I tried some github site that reads the link and generates a QR, but the codes don’t work. This is a complete waste of time.

      • YMS@discuss.tchncs.de
        link
        fedilink
        arrow-up
        5
        ·
        1 year ago

        This is not about turning you in, this is about protecting your users who all possibly just became victims of a crime, and for good reasons it’s not fully upon you to decide whether the possible consequences of this are serious for those users.

      • trouser_mouse@lemmy.world
        link
        fedilink
        arrow-up
        4
        ·
        edit-2
        1 year ago

        It’s more that many people expect those handling their data to be seen to follow the correct procedures and be trusted to handle the data in a fair, transparent, safe and secure way - and in addition to protecting their users, companies are probably encouraged to abide by the regulations because it is very easy for anyone to report where they think action needs to be taken, and regulatory bodies may be more lenient where correct process has been followed.

        If I chance a speeding or parking ticket I can’t be fined nearly 20 million pounds, although I wouldn’t trust some parking companies not to try it! (I’m not saying that would be the case in this instance.)

        https://gdpr.eu/fines/

  • Marek Knápek@lemmy.world
    link
    fedilink
    arrow-up
    76
    arrow-down
    5
    ·
    1 year ago

    So what happened:

    • Someone posted a post.
    • The post contained some instruction to display custom emoji.
    • So far so good.
    • There is a bug in JavaScript (TypeScript) that runs on client’s machine (arbitrary code execution?).
    • The attacker leveraged the bug to grab victim’s JWT (cookie) when the victim visited the page with that post.
    • The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.

    Am I right?

    I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:

    • User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.
    • JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
    • How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
    • The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.

    Am I right? Correct me if I’m wrong.

    Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.

    Marek.

  • dylanTheDeveloper@lemmy.world
    link
    fedilink
    arrow-up
    57
    arrow-down
    6
    ·
    1 year ago

    Good thing we all use randomly generated passwords for every account and always remember to change them every few months.

  • bluemellophone@lemmy.world
    link
    fedilink
    arrow-up
    42
    ·
    edit-2
    1 year ago

    Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?

  • nosut@lemmy.world
    link
    fedilink
    arrow-up
    40
    ·
    edit-2
    1 year ago

    Thanks for the work. As a heads up it appears most of the block instances are back however I believe explodingheads is still missing which you may want to confirm.

    EDIT: it has been added back to the block list.

    • Carnelian@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      1 year ago

      Hey how do you check on that?

      As of the time of me posting this comment, exploding heads is appearing in my feed with some anti lgbt posts. Idk what’s going on because I’m pretty sure they’re supposed to be defederated currently