Updates yarn to the latest version 4.9.2 (from 4.4.0).
Also removes the explicit `enableHardenedMode` from yarnrc as it
significantly slows down installation.
This is already enabled automatically for pull requests on Github, thus
preventing lockfile poisoning where it's relevant.
See <https://yarnpkg.com/features/security#hardened-mode>:
> in most cases you won't even have to think about it - the hardened
mode is enabled by default when Yarn detects it runs in a pull request
from a public GitHub repository.
It can additionally be enabled explicitly for specific CI jobs by using
an environment variable, if desired:
> The hardened mode can be set (or disabled) [...] by defining
`YARN_ENABLE_HARDENED_MODE=1|0` in your environment variables
If this is the case, yarn still recommends **not** enabling it
everywhere:
> **DANGER**
>
> The hardened mode makes installs significantly slower as Yarn has to
query the registry to make sure the information contained in the
lockfile are accurate. If your CI pipeline runs multiple jobs, we
recommend disabling the hardened mode in all but one of them so as to
limit the performance impact.
---------
Co-authored-by: prastoin <paul@twenty.com>
# Introduction
Introducing jest sharding along github actions matrix in order to
fluidify the process
Note in case we will compute and guard coverage metrics we will need to
merge coverages reports such as we do for the frontend storybooks
integrations tests, from now this is overkill as unused
Related successful run
https://github.com/twentyhq/twenty/actions/runs/15585113583/job/43889477889
BlocknoteJS requires an ESM module where our server is CJS, this forced
us to pin the server-util version, which led us to force the resolution
of several packages, leading to bugs downstream.
From Node 22.12 Node supports requiring ESM modules (available from Node
22.0 with a flag). So I upgrade the module.
I picked Node 22 and not Node 23 or Node 24 because 22 is the LTS and we
don't plan to change node versions frequently.
If you remain on Node 18, things should still mostly work, except if you
edit a Rich Text field.
I also starting changing the default runtime for Serverless Functions
which isn't directly related. This means new serverless functions will
be created on Node 22, but we will still need another PR to migrate
existing serverless functions before September (end of support by AWS).
(In this PR I also remove the upgrade commands from 0.43 since they rely
on Blocknote and I didn't want to have to deal with this)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
# Introduction
closes https://github.com/twentyhq/core-team-issues/issues/591
Same than for `twenty-shared` made in
https://github.com/twentyhq/twenty/pull/11083.
## TODO
- [x] Manual migrate twenty-website twenty-ui imports
## What's next:
- Generate barrel and migration script factorization within own package
+ tests
- Refactoring using preconstruct ? TimeBox
- Lint circular dependencies
- Lint import from barrel and forbid them
### Preconstruct
We need custom rollup plugins addition, but preconstruct does not expose
its rollup configuration. It might be possible to handle this using the
babel overrides. But was a big tunnel.
We could give it a try afterwards ! ( allowing cjs interop and stuff
like that )
Stuck to vite lib app
Closed related PRs:
- https://github.com/twentyhq/twenty/pull/11294
- https://github.com/twentyhq/twenty/pull/11203
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
## Context
- Removing search* integration tests instead of fixing them because they
will be replaced by global search very soon
- Fixed billing + add missing seeds to make them work
- Fixed integration tests not using consistently the correct "test" db
- Fixed ci not running the with-db-reset configuration due to nx
configuration being used twice for different level of the command
- Enriched .env.test
- Fixed parts where exceptions were not thrown properly and not caught
by exception handler to convert to 400 when needed
- Refactored feature flag service that had 2 different implementations
in lab and admin panel + added tests
- Fixed race condition when migrations are created at the same timestamp
and doing the same type of operation, in this case object deletion could
break because table could be deleted earlier than its relations
- Fixed many integration tests that were not up to date since the CI has
been broken for a while
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
# Introduction
Applying the most present `yaml` file extension to all
`.github/workflows/` files
## Notes
Regarding the `restore-cache` and `save-cache` composite actions github
agnostically searches for any `DockerFile` `action.yml` and
`action.yaml` within the target folder when invoking composite actions
as follows:
```yml
- name: Restore storybook build cache
if: steps.changed-files.outputs.any_changed == 'true'
uses: ./.github/workflows/actions/restore-cache
with:
key: ${{ env.STORYBOOK_BUILD_CACHE_KEY }}
```
# Introduction
Unless I'm mistaken in the existing `yarn-install` composite action
`dependencies` are cached twice.
## Cached through `setup-node`
As we're providing `yarn` value to cache inputs.
Setup node will cache the global `.yarn`
```yml
- name: Setup Node.js and get yarn cache
uses: actions/setup-node@v3
with:
node-version: 18
cache: yarn
```
## Programmatic `node_modules` caching
We even manually still cache `node_modules` folder using:
```yml
- name: Cache node modules
id: node-modules-cache
uses: actions/cache@v3
with:
path: |
node_modules
packages/*/node_modules
key: root-node_modules-${{ hashFiles('yarn.lock') }}
restore-keys: root-node_modules-
```
By the way it seems like that the `yarn.lock` file is useless as the
restore-key pattern omits it
This could, unless I'm mistaken, leads to invalid cache restore
## Runtime
The current infra results in the following install deps logs:
```sh
Prepare all required actions
Getting action download info
Download action repository 'actions/setup-node@v3' (SHA:1a444[2](https://github.com/twentyhq/twenty/actions/runs/12770942233/job/35596946506#step:6:2)cacd436585916779262731d5b162bc6ec7)
Download action repository 'actions/cache@v[3](https://github.com/twentyhq/twenty/actions/runs/12770942233/job/35596946506#step:6:3)' (SHA:f4b3439a656ba812b8cb417d2d49f9c810103092)
Run ./.github/workflows/actions/yarn-install
Run actions/setup-node@v3
Found in cache @ /opt/hostedtoolcache/node/18.20.5/x6[4](https://github.com/twentyhq/twenty/actions/runs/12770942233/job/35596946506#step:6:4)
Environment details
/usr/local/bin/yarn --version
4.4.0
/usr/local/bin/yarn config get cacheFolder
/home/runner/.yarn/berry/cache
/usr/local/bin/yarn config get enableGlobalCache
true
Cache Size: ~421 MB (441369819 B)
/usr/bin/tar -xf /home/runner/work/_temp/883eae34-9b21-4684-96[5](https://github.com/twentyhq/twenty/actions/runs/12770942233/job/35596946506#step:6:5)8-cdb937f62965/cache.tzst -P -C /home/runner/work/twenty/twenty --use-compress-program unzstd
Cache restored successfully
Cache restored from key: node-cache-Linux-yarn-a8c18b6600f1bc685e60192f39a0a0c5c51b918829090129d3787302789547fc
Run actions/cache@v3
Cache Size: ~506 MB (530305961 B)
/usr/bin/tar -xf /home/runner/work/_temp/becd3767-ac80-4ca9-8d[21](https://github.com/twentyhq/twenty/actions/runs/12770942233/job/35596946506#step:6:23)-93fa61e3070c/cache.tzst -P -C /home/runner/work/twenty/twenty --use-compress-program unzstd
Cache restored successfully
Cache restored from key: root-node_modules-a8c18b6600f1bc685e60192f39a0a0c5c51b918829090129d378730[27](https://github.com/twentyhq/twenty/actions/runs/12770942233/job/35596946506#step:6:30)89547fc
```
Where we can see in details that a cache is hit twice.
```
Cache Size: ~421 MB (441369819 B)
Cache Size: ~506 MB (530305961 B)
```
## Suggestion
We should stick to only one deps caching mechanism.
Also caching the node_modules folder is not
[recommended](611465405c/examples.md (node---npm)).
That's why I would factorize our caching to the `setup-node` scope only.
## The cache-primary-key crafting
Within the cache-primary-key we will inject the following metrics:
- Node version, as we're caching `node_modules` directly in this way to
avoid cache desync in case we upgrade node.
- Hash of the lockfile, in this way if the lockfile comes to mutate we
will re-install the deps from scratch, in this way no need to
programmatically listen for `changed-files` on the `package.json` and
the `lockfile`
- Strict installation with `check-cache` and `enableHardenedMode` to
true and obviously the `--immutable` flag ( note adding the
`--immutable-cache` by principle even if on paper is overkill as a cache
restoration and install should never occurs )
## Introduction
Unless I'm mistaken neither the `inputs.type` or `inputs.type` syntax
exists for `composite actions`.
By default it seems like any `Composite action` inputs are considered as
`string`.
```yml
name: Nx Affected CI
inputs:
parallel:
required: false
types: [number]
default: 3
tag:
required: false
types: [string]
tasks:
required: true
types: [string]
```
## Suggestion
To avoid misunderstanding of our composite actions inputs signature I
would recommend removing any `type` props but also default any inputs as
`strings`
## Misc
- Please find related GitHub community discussion
https://github.com/orgs/community/discussions/65588
## Introduction
In this previous PR https://github.com/twentyhq/twenty/pull/9448 we've
refactored the storybook build caching flow to be using the new
[actions/cache](https://github.com/actions/cache) `restore` and `save`
functions, which significantly improve caching operations duration.
In this way, in this PR, we've standardize both of the `restore` and
`save` by refactoring the `task-cache` composite action. By creating two
new composite actions `save-cache` and `restore-cache` that centralize
the paths to cache and the way to compute the primary key.
## Misc
- **If no cache** is hit, then a job duration will long for its task
duration and nothing else, previously the cache upload would sometimes
take up to 3 mins.
- **if cache** is hit, then mainly the only time consuming step is the
dependencies installation ( which is theory is also cached, in fact
twice. We will be having a look on this issue in an upcoming PR )
- Fixes storybook coverage command: the coverage directory path was
incorrect, but instead of failing `storybook:test --configuration=ci`,
it was hanging indefinitely.
- Switches back to `concurrently` to launch `storybook:static` and
`storybook:test` in parallel, which allows to use options to explicitly
kill `storybook:static` when `storybook:test` fails.
- Moves `storybook:test --configuration=ci` to its own command
`storybook:static:test`: used in the CI, and can be used locally to run
storybook tests without having to launch `storybook:dev` first.
- Creates command `storybook:coverage` and enables cache for this
command.
- Fixes Jest tests that were failing.
- Improves caching conditions for some tasks (for instance, no need to
invalidate Jest test cache if only Storybook story files were modified).
Closes#5097
- Uses "nx affected" to detect what projects need to be checked in the
current PR (for now, `ci-front` and `ci-server` workflows only).
- Caches results of certain tasks (`lint`, `typecheck`, `test`,
`storybook:build`) when a PR pipeline runs. The next runs of the same
PR's pipeline will then be able to reuse the PR's task cache to execute
tasks faster.
- Caches Yarn's cache folder to install dependencies faster in CI jobs.
- Rewrites the node modules cache/install steps as a custom, reusable
Github action.
- Distributes `ci-front` jobs with a "matrix" strategy.
- Sets common tasks config at the root `nx.json`. For instance, to
activate the `typecheck` task in a project, add `typecheck: {}` to its
`project.json` and it'll use the default config set in `nx.json` for the
`typecheck` task. Options can be overridden in each individual
`project.json` if needed.
- Adds "scope" tags to some projects: `scope:frontend`, `scope:backend`,
`scope:shared`. An eslint rule ensures that `scope:frontend` only
depends on `scope:frontent` or `scope:shared` projects, same for
`scope:backend`. These tags are used by `nx affected` to filter projects
by scope and generates different task cache keys according to the
requested scope.
- Enables checks for twenty-emails in the `ci-server` workflow.
