This Pull Request addresses the need to optimize our Continuous
Integration (CI) workflows for Playwright tests and release processes.
The changes implemented aim to reduce unnecessary resource usage by
conditionally executing jobs based on relevant file changes and
Implement https://github.com/tj-actions/changed-files step
## Changes logs
- Updated `ci-test-docker-compose.yaml , ci-chrome-extension.yaml ` to
check for changed files before running tests.
- Updated `ci-front.yaml , ci-utils.yaml , ci-website.yaml ,
ci-server.yaml` to check for changed files before running tests.
- Enhanced `playwright.yml` to skip unnecessary tests based on file
changes.
In this PR, I'm simplifying storybook setup:
1) Remove build --test configuration that prevent autodocs. We are not
using autodocs at all (the dev experience is not good enough), so I have
completely disabled it.
2) Clarify `serve` vs `test` vs `serve-and-test` configurations
After this PR:
- you can serve storybook in two modes: `npx nx run
twenty-front:storybook:serve:dev` and `npx nx run
twenty-front:storybook:serve:static`
- you can run tests agains an already served storybook (this is useful
in dev so you don't have to rebuild everytime to run tests): `npx nx run
twenty-front:storybook:test`
- you can conbine both: `npx nx run
twenty-front:storybook:serve-and-test:static`
TL;DR:
- removed `--configuration={args.scope}` from `storybook:static:test`
for the `storybook:static` part, as it was making `front-sb-test` jobs
in CI not reuse the cache from the `front-sb-build` job and re-build
storybook every time.
- replaced it with a new `test` configuration which optimizes storybook
build for tests and builds storybook 2x faster.
## Fix storybook:build cache usage in CI
`storybook:static:test` executes two scripts in parallel:
1. `storybook:static`, which depends on `storybook:build`
1.a. it builds storybook first with `storybook:build`, the output
directory is `storybook-static`.
1.b. then it launches an `http-server`, using what has been built in
`storybook-static`
2. `storybook:test` to execute tests (needs the storybook http-server to
be running)
When passing `--configuration=pages` or `--configuration=modules` to
`storybook:static` from step 1, those configurations are passed to the
`storybook:build` script from step 1.a as well.
But for Nx `storybook:build` and `storybook:build --configuration=pages`
(or `modules`) are not the same command, therefore one does not reuse
the cache of the other because they could output completely different
things.
As `front-sb-test` jobs are passing `--configuration={args.scope}` to
`storybook:static`, the cache of the previously executed
`storybook:build` (from `front-sb-build`) is not reused and therefore
each job re-builds Storybook with its own scope, which increases CI
time.
### Solution
- Removed scope configurations from `storybook:static` and
`storybook:build` scripts to avoid confusion.
- `storybook:test` and `storybook:dev` can keep scope configurations as
they can be useful and this doesn't impact storybook build cache in CI.
### Improve Storybook build time for testing
Added the `test` configuration to `storybook:build` and
`storybook:static` which makes Storybook build time 2x faster. It
disables addons that slow down build time and are not used in tests.
This PR introduces a Profiling feature for our story book tests.
It also implements a new CI job : front-sb-test-performance, that only
runs stories suffixed with `.perf.stories.tsx`
## How it works
It allows to wrap any component into an array of React Profiler
components that will run tests many times to have the most replicable
average render time possible.
It is simply used by calling the new `getProfilingStory` util.
Internally it creates a defined number of tests, separated by an
arbitrary waiting time to allow the CPU to give more stable results.
It will do 3 warm-up and 3 finishing runs of tests because the first and
last renders are always a bit erratic, so we want to measure only the
runs in-between.
On the UI side it gives a table of results :
<img width="515" alt="image"
src="https://github.com/twentyhq/twenty/assets/26528466/273d2d91-26da-437a-890e-778cb6c1f993">
On the programmatic side, it stores the result in a div that can then be
parsed by the play fonction of storybook, to expect a defined threshold.
```tsx
play: async ({ canvasElement }) => {
await findByTestId(
canvasElement,
'profiling-session-finished',
{},
{ timeout: 60000 },
);
const profilingReport = getProfilingReportFromDocument(canvasElement);
if (!isDefined(profilingReport)) {
return;
}
const p95result = profilingReport?.total.p95;
expect(
p95result,
`Component render time is more than p95 threshold (${p95ThresholdInMs}ms)`,
).toBeLessThan(p95ThresholdInMs);
},
```
- Fixes storybook coverage command: the coverage directory path was
incorrect, but instead of failing `storybook:test --configuration=ci`,
it was hanging indefinitely.
- Switches back to `concurrently` to launch `storybook:static` and
`storybook:test` in parallel, which allows to use options to explicitly
kill `storybook:static` when `storybook:test` fails.
- Moves `storybook:test --configuration=ci` to its own command
`storybook:static:test`: used in the CI, and can be used locally to run
storybook tests without having to launch `storybook:dev` first.
- Creates command `storybook:coverage` and enables cache for this
command.
- Fixes Jest tests that were failing.
- Improves caching conditions for some tasks (for instance, no need to
invalidate Jest test cache if only Storybook story files were modified).
Closes#5097
- Uses "nx affected" to detect what projects need to be checked in the
current PR (for now, `ci-front` and `ci-server` workflows only).
- Caches results of certain tasks (`lint`, `typecheck`, `test`,
`storybook:build`) when a PR pipeline runs. The next runs of the same
PR's pipeline will then be able to reuse the PR's task cache to execute
tasks faster.
- Caches Yarn's cache folder to install dependencies faster in CI jobs.
- Rewrites the node modules cache/install steps as a custom, reusable
Github action.
- Distributes `ci-front` jobs with a "matrix" strategy.
- Sets common tasks config at the root `nx.json`. For instance, to
activate the `typecheck` task in a project, add `typecheck: {}` to its
`project.json` and it'll use the default config set in `nx.json` for the
`typecheck` task. Options can be overridden in each individual
`project.json` if needed.
- Adds "scope" tags to some projects: `scope:frontend`, `scope:backend`,
`scope:shared`. An eslint rule ensures that `scope:frontend` only
depends on `scope:frontent` or `scope:shared` projects, same for
`scope:backend`. These tags are used by `nx affected` to filter projects
by scope and generates different task cache keys according to the
requested scope.
- Enables checks for twenty-emails in the `ci-server` workflow.
# This PR
- Moves dev and ci scripts to the `project.json` file in the
twenty-front package
- Adds a project.json file in the root of the project with the main
start command that start both twenty-server and twenty-front
applications concurrently
- Updates the script command of the root project with the start:prod
command (replacing the start command which will be used in dev with the
help of nx)
- Add a start:prod command in the twenty-front app, replacing the start
command (now used for dev purpose)
Issue ref #4645
@charlesBochet @FelixMalfait please let me know how can I improve it
---------
Co-authored-by: Thaïs Guigon <guigon.thais@gmail.com>
As recommended on the repo of the action we were using
https://github.com/styfle/cancel-workflow-action, we should use
`concurrency.group` rather than a dedicated action.
Here is a PR to use it on all our workflows
Split from https://github.com/twentyhq/twenty/pull/4518
- Upgrades dependencies and applies automatic config migrations with the
command: `npx nx migrate nx` (see
https://nx.dev/nx-api/nx/documents/migrate)
- Fixes lint errors after upgrading `@typescript-eslint`
Note: it was not possible (for now) to migrate Nx to the latest stable
version (v18.2.1) because it upgrades Typescript to v5.4.3, which seems
to cause a bug on install when Yarn tries to apply its native patches.
Might be a bug on the Yarn side.
* wip
* trying to parse display names and emails
* add nodemailer mailparser
* mail parsing is working
* add personId and workspaceMemberId
* add date to messages
* Fix PR
* Run tsc on bigger machine
* Fix lint
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
* Fix CIs
* Fix docs
* Fix eslint-build
* Move file
* Move back
* Fix server ci
* Fix server ci
* Fix server ci
* Fix server ci
* Deactivate e2e tests
* Fix front
* Fix front
* Fix front
* Add twenty-zapier and twenty-utils to the yarn project
* fix
* fix
* Remove pull_request trigger
* merge squashed
- A couple of CJS modules into ESM (config mostly)
- Vite complains about node.js modules: fixed `useIsMatchingLocation.ts`
> or use rollupOptions in vite.config.ts
> ref: f0e4f59d97/vite.config.js (L6)
- Adjust Storybook to work with Vite: use @storybook/test
- Use SWC for jest tranformations
- Remove unused deps:
- ts-jest: replaced with @swc/jest, typecheck by `tsc`
- babel plugins
- @svgr/plugin-jsx: not used
- @testing-library/user-event: handled by @storybook/test
- @typescript-eslint/utils: was not plugged in
- tsup, esbuild-plugin-svgr: will look into that later
- Install Vite required deps, and remove craco/webpack deps
- Adjust SVG to work with Vite as components
- Fixed `Step.tsx`: I dont know if one should be swaped for the other,
but there should be no slash
- Initial formating and linting:
- removed empty object params
- sorting imports, etc..
* prettier: fix pattern
* coverage: sb coverage report isnt working
* Add missing pieces
* `yarn lint --fix`
* fix: scripts permissions
* tsc: cut errors in half
* fix: remove `react-app-env.d.ts`
* tsc: all fixed, except `react-data-grid` types issue
* eslint: ignore env-config.js
* eslint: Align ci with config
* msw: bypass testing warnings
ref: https://stackoverflow.com/questions/68024935/msw-logging-warnings-for-unhandled-supertest-requests
* rebase: and fix things
* Adjust to current `graphql-codegen` no ESM support
* Remove vite plugin and use built-in methods
* rebase: and some fixes
* quick fix + `corepack use yarn@1.22.19`
* Fix build errors
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
* chore: add new eslint config file for ci
* add test console log
* merge the lint steps
* Fix according to PR
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
* Boost CI
* Split CI tests in 2
* Try caching node modules
* Try caching node modules
* Try caching node modules
* Improve CI
* Improve CI
* Improve CI
* Improve CI
* Improve CI
* Improve CI
* Separate jest tests
* Fix typo
* Re-order tests jobs
