**Problem:**
The previous `docker-compose.yml` included a `change-vol-ownership`
service. This service was designed to run once upon startup to `chown`
the `server-local-data` and `docker-data` volumes to user/group
`1000:1000`. This was necessary because:
1. The main `server` and `worker` containers run as the non-root user
`1000` for security.
2. Docker typically creates/mounts named volumes initially owned by
`root`.
3. The application needs write access to these volumes.
However, this run-once service pattern causes problems in certain
deployment environments (like Coolify) that don't gracefully handle
services designed to exit after completing their task. This can lead to
deployment failures or warnings.
**Solution:**
This PR refactors the Docker setup to address the volume permission
issue directly within the Docker image build process, eliminating the
need for the run-once service.
**Changes:**
1. **`packages/twenty-docker/docker-compose.yml`:**
* Removed the `change-vol-ownership` service definition entirely.
* Removed the `depends_on: change-vol-ownership` condition from the
`server` service definition.
* **Proposed Change:** Removed the `${STORAGE_LOCAL_PATH}` environment
variable from the `server-local-data` volume mounts for both `server`
and `worker` services. The path is now hardcoded to
`/app/packages/twenty-server/.local-storage`. (See Reasoning below).
2. **`packages/twenty-docker/twenty/Dockerfile`:**
* In the final stage, *before* the `USER 1000` command, added lines to:
* Create the necessary directories: `RUN mkdir -p
/app/packages/twenty-server/.local-storage /app/docker-data` (and also
`/app/.local-storage` for safety, though it's likely unused by volumes).
* Set the correct ownership: `RUN chown -R 1000:1000 /app/.local-storage
/app/packages/twenty-server/.local-storage /app/docker-data`.
3. **`packages/twenty-docker/twenty/entrypoint.sh`:**
* Added a check near the beginning of the script for the presence of the
now-potentially-unused `STORAGE_LOCAL_PATH` environment variable.
* If the variable is set, a warning message is printed to standard
output, informing the user that the variable might be deprecated and
ignored if the hardcoded path change in `docker-compose.yml` is
accepted.
**Reasoning:**
By creating the target directories
(`/app/packages/twenty-server/.local-storage` and `/app/docker-data`)
within the Docker image *and* setting their ownership to `1000:1000`
during the build (while still running as root), we leverage Docker's
volume initialization behavior. When a named volume is mounted to a
non-empty directory in the container image, Docker copies the content
and ownership from the image directory into the volume. This ensures
that when the `server` and `worker` containers start (running as user
`1000`), the volumes they mount already have the correct permissions,
eliminating the need for the separate `change-vol-ownership` service.
**Regarding `STORAGE_LOCAL_PATH`:**
The `docker-compose.yml` previously allowed configuring the path for
local storage via the `STORAGE_LOCAL_PATH` variable, defaulting to
`.local-storage`. Since the Dockerfile now explicitly creates and sets
permissions for `/app/packages/twenty-server/.local-storage`,
maintaining this configuration might be unnecessary or could potentially
lead to permission errors if a user sets it to a path *not* prepared in
the Dockerfile.
This PR proposes hardcoding the path in `docker-compose.yml` to
`/app/packages/twenty-server/.local-storage` to align with the
Dockerfile changes and simplify configuration. Is this acceptable, or is
there a specific use case for retaining the `STORAGE_LOCAL_PATH`
variable that needs to be considered? If retained, the Dockerfile would
need further changes to dynamically handle permissions based on this
variable.
**Impact:**
* Improves compatibility with deployment platforms that struggle with
run-once containers.
* Simplifies the `docker-compose.yml` setup (potentially, pending
discussion on `STORAGE_LOCAL_PATH`).
* Fixes volume permissions at the source (image build) rather than
relying on a runtime fix.
* Adds a warning for users who might have the potentially deprecated
variable set.
**Testing:**
The changes have been tested locally using `docker compose up`. The
services start correctly, the application is accessible, and the warning
message for the potentially deprecated variable appears as expected when
the variable is set.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Background
I'm trying to self-host twenty using [official 1-Click w/ Docker Compose
guide](https://twenty.com/developers/section/self-hosting/docker-compose)
on AWS EC2 (using `t3.small` instance type).
## What happened
I used the one-line script but it failed with
```
[skipped]
server-1 | Successfuly migrated DB!
dependency failed to start: container twenty-server-1 is unhealthy
```
Then I tried manual steps, and it failed again with same issue as above.
No configuration changed, everything default used.
I re-run manual steps multiple times with `docker compose down -v` and
`docker compose up`, everything time it failed with `server` unhealthy
as it seems server takes longer than configured health check duration
(which is 50 seconds)
Here's the `time` summary running it:
```
[skipped]
server-1 | Successfuly migrated DB!
dependency failed to start: container twenty-server-1 is unhealthy
________________________________________________________
Executed in 58.26 secs fish external
usr time 661.43 millis 362.00 micros 661.07 millis
sys time 646.10 millis 212.00 micros 645.89 millis
root@ip-10-0-10-43 ~/twenty [1]#
```
## Why it happend
My hunch (new to twenty, just used it yesterday) is that server service
takes much longer to become healthy with DB migration and etc, that
configured health check retries is not sufficient.
## What solution worked
Increased the retry for server service from 10 to 20, and it worked and
service came up healthy (everytime). The increase wasn't needed for db
or worker service, just server service.
If this all makes sense, please feel free to merge this, so it'll be
smoother experience for others new to self-hosting twenty.
Adding the placeholders for the environment variables related to setting
up the mail and calendar sync. This will make the Twenty setup easier
for new users.
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Replaced multiple environment variables for frontend URL construction
with a single FRONTEND_URL variable. This change reduces complexity and
improves clarity by consolidating frontend URL handling into one source.
Updated relevant validations and removed unused variables like
FRONT_PROTOCOL and FRONT_PORT.
Fix#10016
This PR updates the docker-compose installation documentation and env
sample to improve the setup experience.
- Updates the URLs for raw files to reference main branch, which is
likely where new users will be pulling from initially. This seems to be
the most straightforward option; assume that advanced users who want to
retrieve it from a particular tag will know to change the URL for their
scenario.
- Fixes an improperly stated curl command.
- Adds a note that the PGPASSWORD_SUPERUSER should be URL-safe. This is
required since the value is later concat into a PG_DATABASE_URL as a
URL, and expected to be in proper URL format. Touches on #8597.
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
See:
https://github.com/twentyhq/twenty/issues/9031#issuecomment-2542015975
I think it would be easier if the default behavior for the container was
to run the migration, and setting the environment variable would be used
to disable it (e.g. on the worker).
Long-term goal is for the default setup to work out of the box with ~2
env variables only (database url, redis url)
I don't think there's a big risk if people forget to turn it off on the
worker?
We will remove the `twenty-postgres` image that was used for local
development and only use `twenty-postgres-pilo` (which we use in prod),
bringing the development environment closer to prod and avoiding having
to maintain 2 images.
Instead of provisioning the super user after the db initialization, we
directly rely on the superuser provided by Spilo for simplicity. We also
introduce a change that tries to create the right database (`default` or
`test`) based on the context.
How to test:
```
docker build -t twentycrm/twenty-postgres-spilo:latest -f ./packages/twenty-docker/twenty-postgres-spilo/Dockerfile .
docker images --no-trunc | grep twenty-postgres-spilo
postgres-on-docker:
docker run \
--name twenty_pg \
-e PGUSER_SUPERUSER=twenty \
-e PGPASSWORD_SUPERUSER=twenty \
-e ALLOW_NOSSL=true \
-v twenty_db_data:/home/postgres/pgdata \
-p 5432:5432 \
REPLACE_WITH_IMAGE_ID
```
Closes#7421
This pull request consolidates Redis connection parameters into a single
`REDIS_URL` environment variable across various configuration files and
code modules. The most important changes include updates to environment
variable files, Docker and Kubernetes configurations, and code modules
to utilize the new `REDIS_URL` format.
### Environment Variable Updates:
*
[`packages/twenty-docker/.env.example`](diffhunk://#diff-f4b5e7acc0dde630eafb2228390ca56bd56db0d183582be6433a9ee808088a4cL8-R8):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-server/.env.example`](diffhunk://#diff-c06e244338b33286ea90221635809785352a971de53f647ea65650124ae74ad2L53-R53):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
*
[`packages/twenty-server/.env.test`](diffhunk://#diff-def44a08e699c9deca2f72c9f87951de5d33d9ccf5621eab2f34978df8ad0954L16-R16):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
### Docker and Kubernetes Configuration Updates:
*
[`packages/twenty-docker/docker-compose.yml`](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL28-R28):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL` in multiple
service definitions.
[[1]](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL28-R28)
[[2]](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL62-R61)
*
[`packages/twenty-docker/k8s/manifests/deployment-server.yaml`](diffhunk://#diff-91623ed4e8b2088947cfa9a5dad76b6013e8db0c150d84347a215fa9ad78bf04L44-R45):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/manifests/deployment-worker.yaml`](diffhunk://#diff-8532debd131ce168a2527fa6a9be6405792178576ee47e2eef0cd9f3ff4a8f8cL43-R44):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/terraform/deployment-server.tf`](diffhunk://#diff-b4c468660ab00bd03589d0d47502c477ca83f7d876ff196534880b02ba46dce0L64-R65):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/terraform/deployment-worker.tf`](diffhunk://#diff-e5d6a1f68d5391e2120aef6261f22f905aa1ce0a567e31b21a27d44f313eaecaL62-R63):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
### Code Module Updates:
*
[`packages/twenty-server/src/engine/core-modules/cache-storage/cache-storage.module-factory.ts`](diffhunk://#diff-06e787a7c8a48022d5909b5df9b8c6ca192521cf32f51d7f561cee937bed6678L23-R35):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-server/src/engine/core-modules/environment/environment-variables.ts`](diffhunk://#diff-26ce615693b053eda02aa48aa2e30400381a2588dcb08d4a9dc3b0bf5bdd6fe7L378-R384):
Added validation for `REDIS_URL`.
*
[`packages/twenty-server/src/engine/core-modules/message-queue/message-queue.module-factory.ts`](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bR6):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
[[1]](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bR6)
[[2]](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bL35-R41)
### Documentation Updates:
*
[`packages/twenty-website/src/content/developers/self-hosting/self-hosting-var.mdx`](diffhunk://#diff-c4cc78a3ce18b6edb10f1aee8990271e1d2796a8c06c1c6ae3b68db8d52278a3L37-R37):
Updated documentation to reflect the change to `REDIS_URL`.
*
[`packages/twenty-website/src/content/developers/self-hosting/upgrade-guide.mdx`](diffhunk://#diff-c7d757829f6128e1f47aa1955bde561292dce558280511fab66281afa99042a5R102-R112):
Added upgrade instructions for the new `REDIS_URL` variable.
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
In this PR:
- update your environment variables to default `CACHE_STORAGE_TYPE` to
`redis` and `MESSAGE_QUEUE_TYPE` to `bull-mq`
- add redis container to our default docker-compose
- add `REDIS_HOST` and `REDIS_PORT` to docker-compose yaml
- add upgrade instructions
Add a new util called `resolveAbsolutePath` to allow providing absolute
path for environment variable like `STORAGE_LOCAL_PATH`.
If the path in the env start with `/` we'll not prefix it with
`process.cwd()`.
Also we're using a static path for the old `db_initialized` file now
named `db_status` and stop using the env variable for this file as this
one shouldn't ne stored in the `STORAGE_LOCAL_PATH`.
Fix#4794
---------
Co-authored-by: Quentin Galliano <qgalliano@gmail.com>
Having 2 different dev setups caused confusion, let's remove the Docker
local setup and recommend people install yarn locally.
Also simplified some docs by merging pages together, the recommend
self-hosting option is now the docker-compose / to adapt the
docker-compose.
