The high CPU issue that is causing midwest.social to slow way down and timeout on HTTP requests is still a mystery to me. But, it seems to do it less often and for less time now.

Here is CPU usage over 24 hours:

It’s a bunch of SELECT statements in Postgres that all seem to fire at one time and take a while to complete as a batch. I’ve inspected the logs and haven’t seen anything unusual. Just stuff federating and posts receiving upvotes.

  • simsym
    link
    51 year ago

    Have you reached out to anyone else who runs a Lemmy server and seen if they are also experiencing this?

    Do you think it is a bottlekneck or just due to large updates from other servers?

    • seahorse [Ohio]OPMA
      link
      41 year ago

      I have reached out and nobody seems to have any idea what would cause it.

      • simsym
        link
        31 year ago

        Not familiar with hosting these servers at all but is there anything seemingly up with the DB? Large table or DB sizes?

        If you want to dump the logs somewhere I can try to take a look. I work in IT as my day job so no promises but I can at least feel my way around if you would like the help. Know this is all out of your own time.

    • seahorse [Ohio]OPMA
      link
      510 months ago

      Unfortunately it still happens, just not nearly as often as it used to. Need to meet up with my database admin to get to the bottom of the issue.

    • RecreationalPlacebos
      link
      410 months ago

      I’ve been having timeout and 404 issues since I created my account last week. Typically can’t browse more than 5 minutes before I start getting timeouts, which can last for several minutes to several hours before anything loads again. Thought it was a gerboa issue at first, but I’m using a browser now and getting them too.

  • @Green_Bay_Guy@lemmy.ml
    link
    fedilink
    410 months ago

    Yo, I’m a network engineer, also from the Midwest (Living in Vietnam). Do you have a discord or chat platform where I could be of any help?

    • seahorse [Ohio]OPMA
      link
      410 months ago

      Yeah, there’s a link to our Matrix chat in the sidebar.

  • Arindrew
    link
    310 months ago

    Has the influx of new members exacerbated this issue? I found this thread because I’m seeing a few ‘504 Gateway Timeout’ errors occasionally.

    • seahorse [Ohio]OPMA
      link
      410 months ago

      Beehaw is also noticing these issues. It’s most likely from the influx of new users.

  • RoundSparrow @ SJW
    link
    fedilink
    2
    edit-2
    9 months ago

    Your image didn’t show up.

    Is it PostgreSQL process that is going high CPU, or the lemmy_server process, or lemmy-ui?

    • seahorse [Ohio]OPMA
      link
      49 months ago

      It’s fixed. Was an inefficient database query.

        • seahorse [Ohio]OPMA
          link
          29 months ago

          I manually modified the activity table. I’ll find the migration that didn’t run.

  • @pelotron
    link
    2
    edit-2
    9 months ago

    Any updates on server stability lately? I’ve been having a lot of connectivity issues the last couple days, and just saw this post on lemmy.world that was pretty interesting.

    @seahorse@midwest.social have you been dealing with performance problems like this?

    (side note, how do you embed a link to another instance that will let midwest.social users stay logged on?)

  • Blake Runnoe
    link
    fedilink
    110 months ago

    @seahorse I haven’t been able to submit my application to join the instance yet (or maybe I have repeatedly, sorry if that’s the case!) is the sign up issue I’m having related to this? Commenting from mastodon, which is kind of amazing that it works.

      • Blake Runnoe
        link
        fedilink
        210 months ago

        @seahorse it’s just spinning for an unusually long time, with no specific error. Got to be 5 minutes by now. I’ve been bouncing between failing the Captcha and getting a “too many requests” error yesterday and earlier today. If it turns out I’m failing the Captcha because I’m a robot, please let me down gently 😉

        • seahorse [Ohio]OPMA
          link
          310 months ago

          It is rate limited by IP address. Close the page, wait 10 minutes, then try again is what I’d do.