For fresh install, we need the migrations to happen before the upgrade
command is triggered as the upgrade command is a NestJS command and the
app will try to load env variables from db
**Problem:**
The previous `docker-compose.yml` included a `change-vol-ownership`
service. This service was designed to run once upon startup to `chown`
the `server-local-data` and `docker-data` volumes to user/group
`1000:1000`. This was necessary because:
1. The main `server` and `worker` containers run as the non-root user
`1000` for security.
2. Docker typically creates/mounts named volumes initially owned by
`root`.
3. The application needs write access to these volumes.
However, this run-once service pattern causes problems in certain
deployment environments (like Coolify) that don't gracefully handle
services designed to exit after completing their task. This can lead to
deployment failures or warnings.
**Solution:**
This PR refactors the Docker setup to address the volume permission
issue directly within the Docker image build process, eliminating the
need for the run-once service.
**Changes:**
1. **`packages/twenty-docker/docker-compose.yml`:**
* Removed the `change-vol-ownership` service definition entirely.
* Removed the `depends_on: change-vol-ownership` condition from the
`server` service definition.
* **Proposed Change:** Removed the `${STORAGE_LOCAL_PATH}` environment
variable from the `server-local-data` volume mounts for both `server`
and `worker` services. The path is now hardcoded to
`/app/packages/twenty-server/.local-storage`. (See Reasoning below).
2. **`packages/twenty-docker/twenty/Dockerfile`:**
* In the final stage, *before* the `USER 1000` command, added lines to:
* Create the necessary directories: `RUN mkdir -p
/app/packages/twenty-server/.local-storage /app/docker-data` (and also
`/app/.local-storage` for safety, though it's likely unused by volumes).
* Set the correct ownership: `RUN chown -R 1000:1000 /app/.local-storage
/app/packages/twenty-server/.local-storage /app/docker-data`.
3. **`packages/twenty-docker/twenty/entrypoint.sh`:**
* Added a check near the beginning of the script for the presence of the
now-potentially-unused `STORAGE_LOCAL_PATH` environment variable.
* If the variable is set, a warning message is printed to standard
output, informing the user that the variable might be deprecated and
ignored if the hardcoded path change in `docker-compose.yml` is
accepted.
**Reasoning:**
By creating the target directories
(`/app/packages/twenty-server/.local-storage` and `/app/docker-data`)
within the Docker image *and* setting their ownership to `1000:1000`
during the build (while still running as root), we leverage Docker's
volume initialization behavior. When a named volume is mounted to a
non-empty directory in the container image, Docker copies the content
and ownership from the image directory into the volume. This ensures
that when the `server` and `worker` containers start (running as user
`1000`), the volumes they mount already have the correct permissions,
eliminating the need for the separate `change-vol-ownership` service.
**Regarding `STORAGE_LOCAL_PATH`:**
The `docker-compose.yml` previously allowed configuring the path for
local storage via the `STORAGE_LOCAL_PATH` variable, defaulting to
`.local-storage`. Since the Dockerfile now explicitly creates and sets
permissions for `/app/packages/twenty-server/.local-storage`,
maintaining this configuration might be unnecessary or could potentially
lead to permission errors if a user sets it to a path *not* prepared in
the Dockerfile.
This PR proposes hardcoding the path in `docker-compose.yml` to
`/app/packages/twenty-server/.local-storage` to align with the
Dockerfile changes and simplify configuration. Is this acceptable, or is
there a specific use case for retaining the `STORAGE_LOCAL_PATH`
variable that needs to be considered? If retained, the Dockerfile would
need further changes to dynamically handle permissions based on this
variable.
**Impact:**
* Improves compatibility with deployment platforms that struggle with
run-once containers.
* Simplifies the `docker-compose.yml` setup (potentially, pending
discussion on `STORAGE_LOCAL_PATH`).
* Fixes volume permissions at the source (image build) rather than
relying on a runtime fix.
* Adds a warning for users who might have the potentially deprecated
variable set.
**Testing:**
The changes have been tested locally using `docker compose up`. The
services start correctly, the application is accessible, and the warning
message for the potentially deprecated variable appears as expected when
the variable is set.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Background
I'm trying to self-host twenty using [official 1-Click w/ Docker Compose
guide](https://twenty.com/developers/section/self-hosting/docker-compose)
on AWS EC2 (using `t3.small` instance type).
## What happened
I used the one-line script but it failed with
```
[skipped]
server-1 | Successfuly migrated DB!
dependency failed to start: container twenty-server-1 is unhealthy
```
Then I tried manual steps, and it failed again with same issue as above.
No configuration changed, everything default used.
I re-run manual steps multiple times with `docker compose down -v` and
`docker compose up`, everything time it failed with `server` unhealthy
as it seems server takes longer than configured health check duration
(which is 50 seconds)
Here's the `time` summary running it:
```
[skipped]
server-1 | Successfuly migrated DB!
dependency failed to start: container twenty-server-1 is unhealthy
________________________________________________________
Executed in 58.26 secs fish external
usr time 661.43 millis 362.00 micros 661.07 millis
sys time 646.10 millis 212.00 micros 645.89 millis
root@ip-10-0-10-43 ~/twenty [1]#
```
## Why it happend
My hunch (new to twenty, just used it yesterday) is that server service
takes much longer to become healthy with DB migration and etc, that
configured health check retries is not sufficient.
## What solution worked
Increased the retry for server service from 10 to 20, and it worked and
service came up healthy (everytime). The increase wasn't needed for db
or worker service, just server service.
If this all makes sense, please feel free to merge this, so it'll be
smoother experience for others new to self-hosting twenty.
The example .env file uses `PG_DATABASE_PASSWORD` so this makes it
consistent. Currently, the podman-compose directions will fail on first
run unless you manually change the env var name.
# Introduction
`upgrade` and `migrate` are not run every time even, but only once on
database creation, tho we're suggesting users they do as not requiring
manual run anymore since `0.50`
close https://github.com/twentyhq/twenty/issues/11671
Let's deprecate Sentry Release and use APP_VERSION instead.
It'll make it more clear in the interface to use named version for bug
analysis, than commit sha
Added files needed to deploy twenty on podman using podman-compose.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
The PR https://github.com/twentyhq/twenty/pull/11400 introduced changes
to the execution permissions of many executable files. These changes
aren't correct and must be reverted.
cc. @charlesBochet
This is a minor rework of PR #10738.
I noticed an inconsistency with how Select options are passed as props.
Many files use constants stored in external files to pass options props
to Select objects. This allows for code reusability. Some files are not
passing options in this format.
I modified more files so that they use this method of passing options
props. I made changes to:
- WorkerQueueMetricsSection.tsx
- SettingsDataModelFieldBooleanForm.tsx
- SettingsDataModelFieldTextForm.tsx
- SettingsDataModelFieldNumberForm.tsx
- PlaygroundSetupForm.tsx
- ViewPickerContentCreateMode.tsx
I also noticed that some of these files were incorrectly using
useLingui(), so I fixed the import and usage where needed.
---------
Co-authored-by: Beau Smith <bsmith26@iastate.edu>
Co-authored-by: Charles Bochet <charles@twenty.com>
A small PR but a big step towards making Twenty easier to self-host and
upgrade!
Now changing the tag and pulling a new version should be the only step
to upgrade as migrations script will be ran automatically upon starting
the containers. It was already the case for typeorm migrations, but not
for standard objects migration and data migration scripts. It is still
possible to disable this behavior for the most complex deployments such
as our own cloud.
Trying to build the images for different build platforms using the
Makefile isn't possible without changing the code.
This change exposes the PLATFORM and TAG variables to enable the user to
build and tag the images with greater flexibility.
The same TAG variable can be used on the prod-*-run targets too.
Postgres is not built in the same way but is run in the Makefile too, so
the TAG variable applied here too, but not the platform.
---------
Co-authored-by: Paul McKeown <paul@midships.io>
# Introduction
This PR contains a big test file and few snapshots
Related to https://github.com/twentyhq/core-team-issues/issues/487
## New env var `APP_VERSION`
Now will be injected directly in a built docker image the twenty's built
version. Inferred from the build git tag name.
Which mean on main or other `not a tag version` built APP_VERSION will
be `null`
## New upgrade-commander-runner
Refactored the upgrade command to be more strict regarding:
- Version management
- Sync metadata command always run
- Added failing workspaces aggregator + logs on cleanup
From now on the `upgrade` command will compare the `WORKSPACE_VERSION`
to the `APP_VERSION` in order to bypass any workspace version != than
the upgrade version `fromVersion`
## Existing commands
Note that the version validation will be done only when passing by the
`upgrade` command.
Which means that running the following command
`upgrade:x.y-some-specific-command` won't result in workspace version
mutation
This is to enforce that all an upgrade commands + sync-metadata has been
run on a workspace
## Will do in other PR but related
### New workspace
New workspace will now be inserted with version equal to the APP_VERSION
they've been created by
### Old workspace
Will create a command that should be ran outside of any `upgrade-runner`
extending command, the command will have to be ran on every workspace
before making the next release upgrade
This command iterates over any active and suspended workspace that has
`version` to `NULL` in order to update it `APP_VERSION` -1 minor
### SENTRY_RELEASE
- Either deprecate SENTRY_RELEASE in favor of `APP_VERSION` => What
about main with null version ? or create a new env var that would be
`APP_COMMIT_SHA` instead of SENTRY third party ref
### Update CD to inject APP_VERSION from branch name
### Update docs and release logs
Adding documentation for `APP_VERSION`
## Related PRs:
https://github.com/twentyhq/twenty-infra/pull/181
The Keystatic's environment variables must be defined during the build.
@prastoin and I identified that the other environment variables aren't
defined during the build. We decided to add fake environment variables
directly in the Dockerfile to make the build pass. Later, the docker
image should be executed with the real environment variables that'll
make Keystatic work properly.
Adding the placeholders for the environment variables related to setting
up the mail and calendar sync. This will make the Twenty setup easier
for new users.
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Replaced multiple environment variables for frontend URL construction
with a single FRONTEND_URL variable. This change reduces complexity and
improves clarity by consolidating frontend URL handling into one source.
Updated relevant validations and removed unused variables like
FRONT_PROTOCOL and FRONT_PORT.
Fix#10016
# Introduction
Defined `dependsOn` for each nx project's configuration that has a
dependency to another local package ( ui, shared ).
As follows:
```json
"dependsOn": ["^build"]
```
Where the `^` symbol means "all dependencies of this project"
Now on a fresh repo, no built or install deps after install dependencies
you can directly hit `npx nx build PROJECT_NAME`
closes https://github.com/twentyhq/core-team-issues/issues/371
Related what was failing
[run](https://github.com/twentyhq/twenty-infra/actions/runs/13141544809/job/36669643182)
Cancelled before deploy, attested build was correct within the publish
and digest
This PR updates the docker-compose installation documentation and env
sample to improve the setup experience.
- Updates the URLs for raw files to reference main branch, which is
likely where new users will be pulling from initially. This seems to be
the most straightforward option; assume that advanced users who want to
retrieve it from a particular tag will know to change the URL for their
scenario.
- Fixes an improperly stated curl command.
- Adds a note that the PGPASSWORD_SUPERUSER should be URL-safe. This is
required since the value is later concat into a PG_DATABASE_URL as a
URL, and expected to be in proper URL format. Touches on #8597.
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
There was a small issue where if you used the 1-click install script
right after it was tagged on Github but before it was built and
published to Docker hub, then it would fail
Update `packages/twenty-docker/k8s/manifests/deployment-db.yaml` file to
match recent changes in docker-compose.yaml version.
- update postgresql docker image to Spilo version
- add important ENV vars
See:
https://github.com/twentyhq/twenty/issues/9031#issuecomment-2542015975
I think it would be easier if the default behavior for the container was
to run the migration, and setting the environment variable would be used
to disable it (e.g. on the worker).
Long-term goal is for the default setup to work out of the box with ~2
env variables only (database url, redis url)
I don't think there's a big risk if people forget to turn it off on the
worker?
### Issue Summary
The `service` name in the Kubernetes manifest for the PostgreSQL
database is defined as `twentycrm-db`. However, the `DSN` configuration
in the Server and Worker `deployments` incorrectly references
`twenty-db`. This mismatch causes the pods to fail during initialization
and crash on boot.
### Fix Description
- Updated the `DSN` values in both the Server and Worker `deployments`
to correctly reference `twentycrm-db`, ensuring alignment with the
database `service` name.
- Verified the deployments to ensure the pods start without issues after
the fix.
### Testing
- Deployed the updated manifests in a development environment.
- Confirmed that both Server and Worker pods initialize correctly and
connect to the database service as expected.
If migration fails for some reason, the script runs anyway and creates
the file which indicates that migration is complete. This is obviously
not cool. If database is not available for some reason and migrations
fail, the container should exit, not continue.
Relevant stack overflow: https://stackoverflow.com/a/2871034
We will remove the `twenty-postgres` image that was used for local
development and only use `twenty-postgres-pilo` (which we use in prod),
bringing the development environment closer to prod and avoiding having
to maintain 2 images.
Instead of provisioning the super user after the db initialization, we
directly rely on the superuser provided by Spilo for simplicity. We also
introduce a change that tries to create the right database (`default` or
`test`) based on the context.
How to test:
```
docker build -t twentycrm/twenty-postgres-spilo:latest -f ./packages/twenty-docker/twenty-postgres-spilo/Dockerfile .
docker images --no-trunc | grep twenty-postgres-spilo
postgres-on-docker:
docker run \
--name twenty_pg \
-e PGUSER_SUPERUSER=twenty \
-e PGPASSWORD_SUPERUSER=twenty \
-e ALLOW_NOSSL=true \
-v twenty_db_data:/home/postgres/pgdata \
-p 5432:5432 \
REPLACE_WITH_IMAGE_ID
```
Closes#7421
This pull request consolidates Redis connection parameters into a single
`REDIS_URL` environment variable across various configuration files and
code modules. The most important changes include updates to environment
variable files, Docker and Kubernetes configurations, and code modules
to utilize the new `REDIS_URL` format.
### Environment Variable Updates:
*
[`packages/twenty-docker/.env.example`](diffhunk://#diff-f4b5e7acc0dde630eafb2228390ca56bd56db0d183582be6433a9ee808088a4cL8-R8):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-server/.env.example`](diffhunk://#diff-c06e244338b33286ea90221635809785352a971de53f647ea65650124ae74ad2L53-R53):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
*
[`packages/twenty-server/.env.test`](diffhunk://#diff-def44a08e699c9deca2f72c9f87951de5d33d9ccf5621eab2f34978df8ad0954L16-R16):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
### Docker and Kubernetes Configuration Updates:
*
[`packages/twenty-docker/docker-compose.yml`](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL28-R28):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL` in multiple
service definitions.
[[1]](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL28-R28)
[[2]](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL62-R61)
*
[`packages/twenty-docker/k8s/manifests/deployment-server.yaml`](diffhunk://#diff-91623ed4e8b2088947cfa9a5dad76b6013e8db0c150d84347a215fa9ad78bf04L44-R45):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/manifests/deployment-worker.yaml`](diffhunk://#diff-8532debd131ce168a2527fa6a9be6405792178576ee47e2eef0cd9f3ff4a8f8cL43-R44):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/terraform/deployment-server.tf`](diffhunk://#diff-b4c468660ab00bd03589d0d47502c477ca83f7d876ff196534880b02ba46dce0L64-R65):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/terraform/deployment-worker.tf`](diffhunk://#diff-e5d6a1f68d5391e2120aef6261f22f905aa1ce0a567e31b21a27d44f313eaecaL62-R63):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
### Code Module Updates:
*
[`packages/twenty-server/src/engine/core-modules/cache-storage/cache-storage.module-factory.ts`](diffhunk://#diff-06e787a7c8a48022d5909b5df9b8c6ca192521cf32f51d7f561cee937bed6678L23-R35):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-server/src/engine/core-modules/environment/environment-variables.ts`](diffhunk://#diff-26ce615693b053eda02aa48aa2e30400381a2588dcb08d4a9dc3b0bf5bdd6fe7L378-R384):
Added validation for `REDIS_URL`.
*
[`packages/twenty-server/src/engine/core-modules/message-queue/message-queue.module-factory.ts`](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bR6):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
[[1]](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bR6)
[[2]](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bL35-R41)
### Documentation Updates:
*
[`packages/twenty-website/src/content/developers/self-hosting/self-hosting-var.mdx`](diffhunk://#diff-c4cc78a3ce18b6edb10f1aee8990271e1d2796a8c06c1c6ae3b68db8d52278a3L37-R37):
Updated documentation to reflect the change to `REDIS_URL`.
*
[`packages/twenty-website/src/content/developers/self-hosting/upgrade-guide.mdx`](diffhunk://#diff-c7d757829f6128e1f47aa1955bde561292dce558280511fab66281afa99042a5R102-R112):
Added upgrade instructions for the new `REDIS_URL` variable.
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
