Disk space getting eaten up by build cache #6726
-
|
We've got a server which has a front-end nextjs application. A back-end flask application, and a postgres db instance all running on a dokku instance. We recently upgraded (3-4 months or so) from an older version of dokku (I think it was 0.25.6) to the latest. At the same time, we upgraded our hosting server on DigitalOcean to a more recent version of Ubuntu to support the latest version of Dokku. We've started running into an issue where we run out of disk space when pushing our apps to our dokku box. We don't recall ever running into this on our older version of dokku. We did have a larger disk drive on the older Ubuntu box (250gb vs 80gb now). But we were running on the old box for almost 3 years without ever running into this issue. And we now have run into this disk space issue after only a few months. We narrowed the issue down to build cache. When running And running Is this normal? It's unclear if it's related to the updated or just coincidental. But it doesn't seem like something that we should have to monitor as closely as we are having to do now. We didn't see any other issues/discussions around this when searching, but it doesn't seem like anything we're doing is different enough that no one else would be running into this too? Any thoughts? |
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 4 replies
-
|
i can confirm the same, the space gets eaten up on each app deploy. i think dokku should be handling this internally or explictly state in the docs to do system prune from time to time |
Beta Was this translation helpful? Give feedback.
-
|
Similar issue here. I see my disk usage from the last two years go up and down, suggesting that Dokku was cleaning up after itself. We upgraded Dokku last December, and the disk usage only goes up unless we use the docker prune commands. This is mostly an issue with our development Dokku (v0.33.9) that hosts our review apps. Those apps are constantly being created and destroyed, but there are a bunch of lingering files in |
Beta Was this translation helpful? Give feedback.
-
|
I wonder if we're getting a bunch of little cache hits because of the new dockerfile image builds that are used to tag images now (and add code etc.). I might be able to get away with adding a |
Beta Was this translation helpful? Give feedback.
-
|
I occasionally run |
Beta Was this translation helpful? Give feedback.
-
|
FWIW, at least in our case it does look like the vast majority of space usage is explicitly build cache. We ran into an issue running |
Beta Was this translation helpful? Give feedback.
-
|
Just wanted to chime on this one - I hit this very recently and it was initially hard to diagnose. Got an error like "command not found" but the actual issue was no disk space remaining. I don't recall exactly which command I ran to clear up space - it was 2 AM and I wanted to go back to sleep. Think it was As a frustrating note, when I tried to deploy with no disk space remaining, there was some weird behavior where Dokku deleted my app's config settings entirely. I'm lucky I had them saved in a .env file locally so I was able to quickly reapply them on the next deploy, but it seems strange. Guessing no disk space leads to Dokku thinking the app no longer exists and deleting stuff, I remember seeing some error message to that effect in CI. I think Dokku should do more to clean up all old docker things (images, build cache, etc). Maybe just include a default cron job that cleans this up once per day, or after every deploy. Would save a lot of pain. NOTE: I am using the Dockerfile deploy method (git:from-image through the github actions plugin) |
Beta Was this translation helpful? Give feedback.
-
|
Oh interesting. @bakatz, it sounds like you hit something very similar to my other post here! #6882 |
Beta Was this translation helpful? Give feedback.
-
Yes, sounds very similar. I tried to look for the log that I saw but it seems Github Actions pruned it. There was definitely a log with something to the effect of "web no longer exists, removing app" which then caused my config to be wiped |
Beta Was this translation helpful? Give feedback.
-
|
Dokku never deletes apps unless you run Dokku runs very few processes in the background:
Dokku otherwise does not delete things on it's own. |
Beta Was this translation helpful? Give feedback.
-
|
As of Dokku 0.35.0, we now run |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for this. I can confirm that after upgrading to ^0.35.0 our disk space resets nightly and we no longer encounter this issue. |
Beta Was this translation helpful? Give feedback.
As of Dokku 0.35.0, we now run
docker builder pruneonce a day at 4:05am server time. This removes all build cache - docker doesn't expose a way to do this on a per-image basis - which may impact dockerfile builds.