# Introduction
Introducing jest sharding along github actions matrix in order to
fluidify the process
Note in case we will compute and guard coverage metrics we will need to
merge coverages reports such as we do for the frontend storybooks
integrations tests, from now this is overkill as unused
Related successful run
https://github.com/twentyhq/twenty/actions/runs/15585113583/job/43889477889
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
## Context
- Removing search* integration tests instead of fixing them because they
will be replaced by global search very soon
- Fixed billing + add missing seeds to make them work
- Fixed integration tests not using consistently the correct "test" db
- Fixed ci not running the with-db-reset configuration due to nx
configuration being used twice for different level of the command
- Enriched .env.test
- Fixed parts where exceptions were not thrown properly and not caught
by exception handler to convert to 400 when needed
- Refactored feature flag service that had 2 different implementations
in lab and admin panel + added tests
- Fixed race condition when migrations are created at the same timestamp
and doing the same type of operation, in this case object deletion could
break because table could be deleted earlier than its relations
- Fixed many integration tests that were not up to date since the CI has
been broken for a while
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
Prepare for better version upgrade system + split admin panel into two
permissions + fix GraphQL generation detection
---------
Co-authored-by: ehconitin <nitinkoche03@gmail.com>
## Context
Integration test ci was not running properly. This PR should fix that
Todo: This was not running for a while, many tests are not succeeding so
I'll try to fix them in this PR now that the action is running properly.
This pull request addresses the issue of ensuring that npx nx run
twenty-front:graphql:generate and npx nx run
twenty-front:graphql:generate --configuration=metadata commands are run
to generate the necessary GraphQL files. This prevents changes from
being missed and ending up in subsequent unrelated PRs.
Changes:
Added a step in the ci-server.yml workflow to check for pending GraphQL
generation.
If any GraphQL changes are detected, the CI will fail, and an error
message will be displayed instructing the developer to run the necessary
commands and commit the changes.
This approach is similar to the existing TypeORM migration check and
helps maintain consistency and correctness in the codebase.
Issue Resolved: #9726
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Motivations
This is not because we've hit the `actions/cache/restore` cache that NX
won't be re-building sub commands.
Which means the grain to save or not the cache should be extracted from
nx sub commands and not from the `actions/cache` outputs.
We should investigate on the way to retrieve this granularity level
From the moment we will be saving everything, which could result
sometimes in duplicating cache instances.
We should keep in mind that our cache occurrences have a 1 day retention
date, is low cost and pretty fast to perform
## Introduction
In this previous PR https://github.com/twentyhq/twenty/pull/9448 we've
refactored the storybook build caching flow to be using the new
[actions/cache](https://github.com/actions/cache) `restore` and `save`
functions, which significantly improve caching operations duration.
In this way, in this PR, we've standardize both of the `restore` and
`save` by refactoring the `task-cache` composite action. By creating two
new composite actions `save-cache` and `restore-cache` that centralize
the paths to cache and the way to compute the primary key.
## Misc
- **If no cache** is hit, then a job duration will long for its task
duration and nothing else, previously the cache upload would sometimes
take up to 3 mins.
- **if cache** is hit, then mainly the only time consuming step is the
dependencies installation ( which is theory is also cached, in fact
twice. We will be having a look on this issue in an upcoming PR )
Solves [ https://github.com/twentyhq/private-issues/issues/214 ]
**TLDR**
Add unit and integration tests to Billing. First approach to run jest
integration tests directly from VSCode.
**In order to run the unit tests:**
Run unit test using the CLI or with the jest extension directly from
VSCode.
**In order to run the integration tests:**
Ensure that your database has the billingTables. If that's not the case,
migrate the database with IS_BILLING_ENABLED set to true:
` npx nx run twenty-server:test:integration
test/integration/billing/suites/billing-controller.integration-spec.ts`
**Doing:**
- Unit test on transformSubscriptionEventToSubscriptionItem
- More tests cases in billingController integration tests.
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
We will remove the `twenty-postgres` image that was used for local
development and only use `twenty-postgres-pilo` (which we use in prod),
bringing the development environment closer to prod and avoiding having
to maintain 2 images.
Instead of provisioning the super user after the db initialization, we
directly rely on the superuser provided by Spilo for simplicity. We also
introduce a change that tries to create the right database (`default` or
`test`) based on the context.
How to test:
```
docker build -t twentycrm/twenty-postgres-spilo:latest -f ./packages/twenty-docker/twenty-postgres-spilo/Dockerfile .
docker images --no-trunc | grep twenty-postgres-spilo
postgres-on-docker:
docker run \
--name twenty_pg \
-e PGUSER_SUPERUSER=twenty \
-e PGPASSWORD_SUPERUSER=twenty \
-e ALLOW_NOSSL=true \
-v twenty_db_data:/home/postgres/pgdata \
-p 5432:5432 \
REPLACE_WITH_IMAGE_ID
```
Run the CI integrationin sync mode
and add the option to run it without db reset
cleaning all the useless integration test
---------
Co-authored-by: guillim <guillaume@twenty.com>
TLDR:
Secure connexion between tinybird and twenty using jwt when accessing
datasource from tinybird.
Solves:
https://github.com/twentyhq/private-issues/issues/73
In order to test:
1. Set ANALYTICS_ENABLED to true
2. Set TINYBIRD_JWT_TOKEN to the ADMIN token from the workspace
twenty_analytics_playground
3. Set TINYBIRD_JWT_TOKEN to the datasource or your admin token from the
workspace twenty_analytics_playground
4. Create a Webhook in twenty and set wich events it needs to track
5. Run twenty-worker in order to make the webhooks work.
6. Do your tasks in order to populate the data
7. Enter to settings> webhook>your webhook and the statistics section
should be displayed.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
This Pull Request addresses the need to optimize our Continuous
Integration (CI) workflows for Playwright tests and release processes.
The changes implemented aim to reduce unnecessary resource usage by
conditionally executing jobs based on relevant file changes and
Implement https://github.com/tj-actions/changed-files step
## Changes logs
- Updated `ci-test-docker-compose.yaml , ci-chrome-extension.yaml ` to
check for changed files before running tests.
- Updated `ci-front.yaml , ci-utils.yaml , ci-website.yaml ,
ci-server.yaml` to check for changed files before running tests.
- Enhanced `playwright.yml` to skip unnecessary tests based on file
changes.
### Summary
This PR introduces several integration tests, a mix of manually written
tests and those generated using the `generate-integration-tests` Python
script located in the `scripts` folder.
### Tests Added:
- **Authentication tests**: Validating login, registration, and token
handling.
- **FindMany queries**: Fetching multiple records for all existing
entities that do not require input arguments.
### How the Integration Tests Work:
- A `setupTest` function is called during the Jest test run. This
function initializes a test instance of the application and exposes it
on a dedicated port.
- Since tests are executed in isolated workers, they do not have direct
access to the in-memory app instance. Instead, the tests query the
application through the exposed port.
- A static accessToken is used, this one as a big expiration time so it
will never expire (365 years)
- The queries are executed, and the results are validated against
expected outcomes.
### Current State and Next Steps:
- These tests currently run using the existing development seed data. We
plan to introduce more comprehensive test data using `faker` to improve
coverage.
- At the moment, the only mutation tests implemented are for
authentication. Future updates should include broader mutation testing
for other entities.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Closes#5097
- Uses "nx affected" to detect what projects need to be checked in the
current PR (for now, `ci-front` and `ci-server` workflows only).
- Caches results of certain tasks (`lint`, `typecheck`, `test`,
`storybook:build`) when a PR pipeline runs. The next runs of the same
PR's pipeline will then be able to reuse the PR's task cache to execute
tasks faster.
- Caches Yarn's cache folder to install dependencies faster in CI jobs.
- Rewrites the node modules cache/install steps as a custom, reusable
Github action.
- Distributes `ci-front` jobs with a "matrix" strategy.
- Sets common tasks config at the root `nx.json`. For instance, to
activate the `typecheck` task in a project, add `typecheck: {}` to its
`project.json` and it'll use the default config set in `nx.json` for the
`typecheck` task. Options can be overridden in each individual
`project.json` if needed.
- Adds "scope" tags to some projects: `scope:frontend`, `scope:backend`,
`scope:shared`. An eslint rule ensures that `scope:frontend` only
depends on `scope:frontent` or `scope:shared` projects, same for
`scope:backend`. These tags are used by `nx affected` to filter projects
by scope and generates different task cache keys according to the
requested scope.
- Enables checks for twenty-emails in the `ci-server` workflow.
As recommended on the repo of the action we were using
https://github.com/styfle/cancel-workflow-action, we should use
`concurrency.group` rather than a dedicated action.
Here is a PR to use it on all our workflows
* Fix CIs
* Fix docs
* Fix eslint-build
* Move file
* Move back
* Fix server ci
* Fix server ci
* Fix server ci
* Fix server ci
* Deactivate e2e tests
* Fix front
* Fix front
* Fix front
* Add twenty-zapier and twenty-utils to the yarn project
* fix
* fix
* Remove pull_request trigger
* feat: wip e2e server test
* feat: use github action postgres & use infra for local
* feat: company e2e test
* feat: add company e2e test for permissions
* Simplify server e2e test run
* Fix lint
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
