Suspected memory leak in webapp (Docker environment)​​

[daimajia]

I deployed the v4 beta in a Docker environment. I ​did not set up or deploy any new projects. After deployment, I ​left it idle without any operations, but monitoring showed that the webapp's memory usage ​continuously increased. At a certain point, I ​restarted the webapp and all applications, yet the webapp's memory ​still continued to rise afterwards. I also encountered a ​similar issue in v3. I ​suspect this might be a memory leak.

[daimajia]

Thank you to the Trigger.dev team for their open-source product. I've been using it from v2 through v3 and am currently testing v4. However, I feel that the current direction of Trigger somewhat deviates from the core principles of technology. In pursuit of a polished UI and modular manageability, it sacrifices performance, significantly increasing costs. After self-hosting v3, I noticed that a large portion of resources is consumed by container management rather than the user code itself. Even the simplest tasks incur substantial overhead due to starting and stopping containers. I'm not sure if this is due to my deployment setup (though I personally doubt it), but I believe the current technical approach is deeply flawed. Tasks that could be handled by a few dozen threads are instead spun up into dozens of containers with unnecessary overhead. I hope this can spark more discussion, as I genuinely love this product. If I've misunderstood or gotten anything wrong, please feel free to point it out.

[nicktrn]

Hey @daimajia thanks for the honest review!

You should be able to solve the webapp memleak issue by setting NODE_MAX_OLD_SPACE_SIZE, there are examples of values here:

trigger.dev/hosting/docker/.env.example

Lines 50 to 59 in 9ecda81

	# Webapp - memory management
	# - This sets the maximum memory allocation for Node.js heap in MiB (e.g. "4096" for 4GB)
	# - It should be set according to your total webapp machine's memory or any container limits you have set
	# - Setting this too high or low WILL cause crashes, inefficient memory utilization and high CPU usage
	# - You should allow for some memory overhead, we suggest at least 20%, for example:
	# - 2GB machine: NODE_MAX_OLD_SPACE_SIZE=1600
	# - 4GB machine: NODE_MAX_OLD_SPACE_SIZE=3200
	# - 6GB machine: NODE_MAX_OLD_SPACE_SIZE=4800
	# - 8GB machine: NODE_MAX_OLD_SPACE_SIZE=6400
	# NODE_MAX_OLD_SPACE_SIZE=8192

Make sure you're also using the latest compose file so it's correctly passed to the webapp: https://github.com/triggerdotdev/trigger.dev/blob/main/hosting/docker/webapp/docker-compose.yml

Even the simplest tasks incur substantial overhead due to starting and stopping containers.

You're definitely not wrong there. We addressed this issue in our Cloud product by adding warm starts, which essentially keep the containers around for a while to try and execute any further runs in sequence.

When self-hosting, one suggestion would definitely be to batch work together into a single run if possible. This would avoid unnecessary container start overheads as you'd only pay the cost once. This is particularly important for very fast runs.

There is still the additional limitation of being unable to execute runs inside a single container in parallel due to how container checkpoints currently work. But we have plans to address this and have some good ideas of how to solve this.