I recently had to expand my instance’s storage size from 25 GB to 50 GB because it was out of space. How much storage do you all use for your instances? I’m a little surprised that it ran out of space as my instance isn’t heavily used.

Am I doing something wrong? I prune all the unused docker images, so it’s not that.

@dessalines@lemmy.ml how much space does lemmy.ml use?

Dessalines
link
fedilink
7
edit-2
2M

Pretty much nothing, postgres volumes folder is ˜500MB, but that really doesn’t change much. Remember that text data really doesn’t take up much space: all of english wikipedia is about 20 GB.

Picture storage in pictrs is the main issue. The only thing I could suggest there, is to use either a remotely running pictrs, or to use sshfs to mount the pictrs volumes folder to a remote machine (non-VPS) that has a ton of storage.

You could also delete old pictures. I think mastodon does this.

I’m not familiar with pictrs and how it works, but software I wrote which contains thumbnails and full copies of every image on the fediverse along with fingerprints, full database metadata, etc, doesn’t take up even a tiny fraction of that space. So…

mastodon only does this for remote content!

TIL! What’s your mastodon acct username?

sorry I am not saying, I want to keep the identities separate.

Same deal here lol

good! It’s better to be safe than sorry ^^

poVoq
link
fedilink
22M

Would it be possible to add some client side code to downscale and compless pictures before upload? Doing it on the server can be quite resource intensive…

Dessalines
link
fedilink
22M

I really don’t want to add a photo editing suite to lemmy-ui. Picture and video processing is something that should be done as much as possible on the server.

I do

  • remove pictrs temp files /tmp/magick-*
  • recompress overly large image files
  • truncate docker logs which frees up a lot of space

recompress overly large image files

When I was looking at some very old posts from feddit.de they had terrible jpg compression artifacts, is this the reason?

Wow yeah that is pretty fast growth!

@wintermute@feddit.de how much space does our server use?

Right now 13GB

@seahorse
admin
creator
link
fedilink
62M

I just saw in the Lemmy matrix chat that you cleared up a lot of space by truncating logs. Where are those?

you could truncate -s 0 /var/lib/docker/containers/*/*-json.log (which might break logs currently being written), configure logging via docker-compose.yml, or even better /etc/docker/daemon.json

https://github.com/LemmyNet/lemmy-ansible/pull/49

@seahorse
admin
creator
link
fedilink
52M

Thanks. I’ll have to look up how to configure daemon.json. I just truncated the logs for now. I wish I had known about that earlier! It cleared up about 15 GB of space.

poVoq
link
fedilink
6
edit-2
2M

Oh, wow. I guess the default log settings should be much less verbose?

Redpandalovely
admin
link
fedilink
42M

Wow, logs take up a ridiculous amount of space!

you could also free up space on pictrs docker exec -it YOUR_PICTRS_CONTAINER /bin/sh -c "rm /tmp/magick-*"

@seahorse
admin
creator
link
fedilink
22M

Hmm, tried that. It said no magick-* files exist.

hm, maybe this problem has been fixed

@seahorse
admin
creator
link
fedilink
42M

I created this instance with docker directly instead of using Ansible. Not sure if that could be why.

okay that’s not too bad.

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.

  • 0 users online
  • 1 user / day
  • 11 users / week
  • 36 users / month
  • 195 users / 6 months
  • 4 subscribers
  • 220 Posts
  • 1.73K Comments
  • Modlog