Another day, another update.

More troubleshooting was done today. What did we do:

  • Yesterday evening @phiresky@phiresky@lemmy.world did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
  • @cetra3@lemmy.ml created a docker image containing 3PR’s: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
  • We started using this image, and saw a big drop in CPU usage and disk load.
  • We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
  • We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
  • We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set proxy_next_upstream timeout; max_fails=5 in nginx.

Currently we’re running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the proxy_next_upstream timeout; max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @phiresky@lemmy.world , @cetra3@lemmy.ml , @stanford@discuss.as200950.com, @db0@lemmy.dbzer0.com , @jelloeater85@lemmy.world , @TragicNotCute@lemmy.world for their help!

And not to forget, thanks to @nutomic@lemmy.ml and @dessalines@lemmy.ml for their continuing hard work on Lemmy!

And thank you all for your patience, we’ll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that’s now started, and I noticed the proxy_next_upstream timeout setting didn’t work (or I didn’t set it properly) so I used max_fails=5 for each upstream, that does actually work.

  • MetricExpansion@lemmy.world
    link
    fedilink
    arrow-up
    21
    ·
    1 year ago

    I’m very curious: does single Lemmy instance have the ability to horizontally scale to multiple machines? You can only get so big of a machine. You did mention a second container, so that would suggest that the Lemmy software is able to do so, but I’m curious if I’m reading that right.

    • DoomBot5@lemmy.world
      link
      fedilink
      arrow-up
      18
      ·
      1 year ago

      A single instance, no. You run multiple instances on multiple machines, then put a frontend (nginx in this case) to distribute the traffic among them.

      • MetricExpansion@lemmy.world
        link
        fedilink
        arrow-up
        6
        ·
        1 year ago

        Interesting, how does that actually work then? Are they just sharing the same database? Is that a supported configuration?

        • T156@lemmy.world
          link
          fedilink
          arrow-up
          11
          ·
          1 year ago

          They would all have to share a database, although I don’t see why there would be any issues.

          The database software they’re using can fairly comfortably handle multiple changes at once, although it might get a bit funny if a user is split between multiple instances.

          • sylverstream@lemmy.nz
            link
            fedilink
            arrow-up
            6
            ·
            1 year ago

            Usually in such cases you would need sticky sessions, i.e. a user always stays at the same frontend.

            • AssPennies@lemmy.world
              link
              fedilink
              English
              arrow-up
              4
              ·
              1 year ago

              If you want to get really fancy, could do some distributed session management with something like redis. Though the underlying application needs to be abstracted out already to allow for it.

        • phiresky@lemmy.world
          link
          fedilink
          arrow-up
          2
          ·
          1 year ago

          it’s not really officially supported, but both lemmy.world and lemm.ee are running with this configuration (multiple lemmy-ui and lemmy_server instances, one pg database).