## Context
Add an eventEmitter instance to twenty datasources so we can emit DB
events.
Add input and output formatting to twenty orm (formatData, formatResult)
Those 2 elements simplified existing logic when we interact with the
ORM, input will be formatted by the ORM so we can directly use
field-like structure instead of column-like. The output will be
formatted, for builder queries it will be in `result.generatedMaps`
where `result.raw` preserves the previous column-like structure.
Important change: We now have an authContext that we can pass when we
get a repository, this will be used for the different events emitted in
the ORM. We also removed the caching for repositories as it was not
scaling well and not necessary imho
Note: An upcoming PR should handle the onDelete: cascade behavior where
we send DESTROY events in cascade when there is an onDelete: CASCADE on
the FK.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Context :
- Phones import is a bit complex if not all subfields are provided.
- Phones subfield validation are absent or different from BE validation.
Solution :
- Normalize callingCode and countryCode validation (BE/FE)
- Ease phone import if only phoneNumber is provided
This PR fixes a bug with phone input clearing its value when we press
space right after a country calling code.
As the problem comes from the library `react-phone-input-number` this PR
implements a yarn patch.
Fixes https://github.com/twentyhq/twenty/issues/12903
### Added IMAP integration
This PR adds support for connecting email accounts via IMAP protocol,
allowing users to sync their emails without OAuth.
#### DB Changes:
- Added customConnectionParams and connectionType fields to
ConnectedAccountWorkspaceEntity
#### UI:
- Added settings pages for creating and editing IMAP connections with
proper validation and connection testing.
- Implemented reconnection flows for handling permission issues.
#### Backend:
- Built ImapConnectionModule with corresponding resolver and service for
managing IMAP connections.
- Created MessagingIMAPDriverModule to handle IMAP client operations,
message fetching/parsing, and error handling.
#### Dependencies:
Integrated `imapflow` and `mailparser` libraries with their type
definitions to handle the IMAP protocol communication.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
BlocknoteJS requires an ESM module where our server is CJS, this forced
us to pin the server-util version, which led us to force the resolution
of several packages, leading to bugs downstream.
From Node 22.12 Node supports requiring ESM modules (available from Node
22.0 with a flag). So I upgrade the module.
I picked Node 22 and not Node 23 or Node 24 because 22 is the LTS and we
don't plan to change node versions frequently.
If you remain on Node 18, things should still mostly work, except if you
edit a Rich Text field.
I also starting changing the default runtime for Serverless Functions
which isn't directly related. This means new serverless functions will
be created on Node 22, but we will still need another PR to migrate
existing serverless functions before September (end of support by AWS).
(In this PR I also remove the upgrade commands from 0.43 since they rely
on Blocknote and I didn't want to have to deal with this)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
# Introduction
Diff description: ~500 tests and +500 additions
close https://github.com/twentyhq/core-team-issues/issues/731
## What has been done here
In a nutshell on a field metadata type ( `SELECT MULTI_SELECT` ) update,
we will be browsing all `ViewFilters` in a post hook searching for some
referencing related updated `fieldMetadata` select. In order to update
or delete the `viewFilter` depending on the associated mutations.
## How to test:
- Add FieldMetadata `SELECT | MULTI_SELECT` to an existing or a new
`objectMetadata`
- Create a filtered view on created `fieldMetadata` with any options you
would like
- Remove some options ( in the best of the world some that are selected
by the filter ) from the `fieldMetadata` settings page
- Go back to the filtered view, removed or updated options should have
been hydrated in the `displayValue` and the filtered data should make
sense
## All filtered options are deleted edge case
If an update implies that a viewFilter does not have any existing
related options anymore, then we remove the viewFilter
## Testing
```sh
PASS test/integration/metadata/suites/field-metadata/update-one-field-metadata-related-record.integration-spec.ts (27 s)
update-one-field-metadata-related-record
SELECT
✓ should delete related view filter if all select field options got deleted (2799 ms)
✓ should update related multi selected options view filter (1244 ms)
✓ should update related solo selected option view filter (1235 ms)
✓ should handle partial deletion of selected options in view filter (1210 ms)
✓ should handle reordering of options while maintaining view filter values (1487 ms)
✓ should handle no changes update of options while maintaining existing view filter values (1174 ms)
✓ should handle adding new options while maintaining existing view filter (1174 ms)
✓ should update display value with options label if less than 3 options are selected (1249 ms)
✓ should throw error if view filter value is not a stringified JSON array (1300 ms)
MULTI_SELECT
✓ should delete related view filter if all select field options got deleted (1127 ms)
✓ should update related multi selected options view filter (1215 ms)
✓ should update related solo selected option view filter (1404 ms)
✓ should handle partial deletion of selected options in view filter (1936 ms)
✓ should handle reordering of options while maintaining view filter values (1261 ms)
✓ should handle no changes update of options while maintaining existing view filter values (1831 ms)
✓ should handle adding new options while maintaining existing view filter (1610 ms)
✓ should update display value with options label if less than 3 options are selected (1889 ms)
✓ should throw error if view filter value is not a stringified JSON array (1365 ms)
Test Suites: 1 passed, 1 total
Tests: 18 passed, 18 total
Snapshots: 18 passed, 18 total
Time: 27.039 s
```
## Out of scope
- We should handle ViewFilter validation when extracting its definition
from the metadata
https://github.com/twentyhq/core-team-issues/issues/1009
## Concerns
- Are we able through the api to update an RATING fieldMetadata ? ( if
yes than that's an issue and we should handle RATING the same way than
for SELECT and MULTI_SELECT )
- It's not possible to group a view from a MULTI_SELECT field
The above points create a double nor a triple "lecture" to the post hook
effect:
- ViewGroup -> only SELECT
- VIewFilter -> only SELECT || MULTI_SELECT
- Rating nothing
I think we should determine the scope of all of that
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
This PR changes the way we do automatching in the import feature.
It uses [Fuse.js](https://www.fusejs.io/) to do a fuzzy text search on
fields and sub-fields.
The labels of sub-fields are now derived from the common config constant
we have for sub-fields.
This PR removes use-context-selector completely, so that any bug
associated with state synchronization between recoil and
use-context-selector disappears.
There might be a slight performance decrease on the table, but since we
have already improved the average performance per line by a lot, and
that the performance bottleneck right now is the fetch more logic and
the windowing solution we use, it is not relevant.
Also the DX has become so hindered by this parallel state logic recently
(think [cache
invalidation](https://martinfowler.com/bliki/TwoHardThings.html)), that
the main benefit we gain from this removal is the DX improvement.
Fixes https://github.com/twentyhq/twenty/issues/12123
Fixes https://github.com/twentyhq/twenty/issues/12109
related to https://github.com/twentyhq/core-team-issues/issues/601
## Done
- add a `onDbEvent` `Subscription` graphql endpoint to listen to
database_event using what we have done with webhooks:
- you can subscribe to any `action` (created, updated, ...) for any
`objectNameSingular` or a specific `recordId`. Parameters are nullable
and treated as wildcards when null.
- returns events with following shape
```typescript
@Field(() => String)
eventId: string;
@Field()
emittedAt: string;
@Field(() => DatabaseEventAction)
action: DatabaseEventAction;
@Field(() => String)
objectNameSingular: string;
@Field(() => GraphQLJSON)
record: ObjectRecord;
@Field(() => [String], { nullable: true })
updatedFields?: string[];
```
- front provide a componentEffect `<ListenRecordUpdatesEffect />` that
listen for an `objectNameSingular`, a `recordId` and a list of
`listenedFields`. It subscribes to record updates and updates its apollo
cached value for specified `listenedFields`
- subscription is protected with credentials
## Result
Here is an application with `workflowRun`
https://github.com/user-attachments/assets/c964d857-3b54-495f-bf14-587ba26c5a8c
---------
Co-authored-by: prastoin <paul@twenty.com>
Recoil-sync was causing issues with Firefox, replacing it with a simpler
mechanism to hydrate variables on page load
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
## What
- Deprecate overlayscrollbars as we decided to follow the native
behavior
- rework on performances (avoid calling recoil states too much at field
level which is quite expensive)
- Also implements:
https://github.com/twentyhq/core-team-issues/issues/569
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
# Introduction
closes https://github.com/twentyhq/core-team-issues/issues/591
Same than for `twenty-shared` made in
https://github.com/twentyhq/twenty/pull/11083.
## TODO
- [x] Manual migrate twenty-website twenty-ui imports
## What's next:
- Generate barrel and migration script factorization within own package
+ tests
- Refactoring using preconstruct ? TimeBox
- Lint circular dependencies
- Lint import from barrel and forbid them
### Preconstruct
We need custom rollup plugins addition, but preconstruct does not expose
its rollup configuration. It might be possible to handle this using the
babel overrides. But was a big tunnel.
We could give it a try afterwards ! ( allowing cjs interop and stuff
like that )
Stuck to vite lib app
Closed related PRs:
- https://github.com/twentyhq/twenty/pull/11294
- https://github.com/twentyhq/twenty/pull/11203
# Introduction
Lately encountering a lot of out of memory error when running
twenty-front in watch mode with both TypeScript and lint checkers
```ts
Error: Worker terminated due to reaching memory limit: JS heap out of memory
at new NodeError (node:internal/errors:405:5)
at [kOnExit] (node:internal/worker:287:26)
at Worker.<computed>.onexit (node:internal/worker:209:20)
```
The existing configuration looks like this:
```ts
// packages/twenty-front/vite.config.ts
'cd ../.. && eslint packages/twenty-front --report-unused-disable-directives --max-warnings 0 --config .eslintrc.cjs',
```
This is wrong as computing the root eslintrc completely omitting
twenty-front's one ***and its ignorePattern*** so will be checking in
`node_modules` etc checking for project-structure :).
For example this a
[snippet](https://gist.github.com/prastoin/d7f8ad4ef5eb2f7732209b756a38094c)
of the above commands errors. We can see rule that should be disabled by
`eslintrc.react.cjs` extension made from twenty-front `eslintrc` :
```ts
/Users/paulrastoin/ws/twenty-two/packages/twenty-front/src/modules/settings/data-model/fields/forms/components/__stories__/SettingsDataModelFieldSettingsFormCard.stories.tsx
23:27 warning Forbidden non-null assertion @typescript-eslint/no-non-null-assertion
```
## Fixes
- consume the `twenty-front` package eslint configuration within the
vite lint checker
- eslint overrides extends are getting merged based on glob inclusion of
their files declarations
- any linted files should be included in one of our `tsconfig`
- removed redundant and counter-productive negative `ignorePatterns`, as
eslint will naturally only lint files within configuration file
directory by default which will result making it go through local
`node_modules` project structure
## Now
Less cpu usage <3.5 gb and faster
```ts
// from packages/twenty-front
TIMING=1 npx eslint . --report-unused-disable-directives --max-warnings 0 --config .eslintrc.cjs --debug
#...
Rule | Time (ms) | Relative
:-----------------------------------------------|----------:|--------:
project-structure/folder-structure | 19578.927 | 20.2%
prettier/prettier | 13746.156 | 14.2%
no-redeclare | 9546.570 | 9.9%
@nx/workspace-explicit-boolean-predicates-in-if | 8167.805 | 8.4%
@typescript-eslint/no-unused-vars | 6872.803 | 7.1%
import/no-relative-packages | 6577.273 | 6.8%
@nx/enforce-module-boundaries | 6520.945 | 6.7%
import/no-duplicates | 4987.476 | 5.2%
react/no-direct-mutation-state | 2323.082 | 2.4%
react/require-render-return | 1155.261 | 1.2%
```
## Conclusion
Please note that `nx linter` might not be as strict as vite config
eslint runner
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
Done :
- move metrics and health cache services from health module to metrics
module
- refactor metrics counter from specific method to set up from enum keys
- add OpenTelemetry (Otel) instrumentation for metrics
- set up Otel SDK to send metrics to Otel collector
To do later :
- implement Otel instrumentation for traces + plug Sentry on top
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
In this PR, I'm specifying to vite build that
'@scalar/api-reference-react' is an external dependency and should not
be considered as a module we maintain (it won't get its own chunk at
build time and we won't generate sourcemaps on our end).
I'm not sure why vite is considering it internal in the first place (I
can see that it's generating .vue.js files, might be the first time we
are relying on a vue library)
This PR introduces Keystatic to let us edit twenty.com's content with a
CMS. For now, we'll focus on creating release notes through Keystatic as
it uses quite simple Markdown. Other types of content will need some
refactoring to work with Keystatic.
https://github.com/user-attachments/assets/e9f85bbf-daff-4b41-bc97-d1baf63758b2
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
## Introduction
This is PR is a suggestion ! And should be discussed
With yarn `^4`, during installation won't raise an error if current dev
env does not satisfies the `engines` policy.
We have usually 10+ contributors support request regarding higher node
version issue per week
I would have preferred a very declarative integration using npm
[engines](https://docs.npmjs.com/cli/v11/configuring-npm/package-json#engines)
but this does not seems to be natively supported by `yarn`
We should keep in mind that this might block any machines from our CICD
if they have diff node version installed ( such as running the project
on a different node version could result in bugs too )
## Implem
Created a yarn [constraints](https://yarnpkg.com/features/constraints)
run after each installation that checking if current node version
satisfies defined engines range ( might also be done for others engines
entries )
I assume we will always have the same engines policy for every packages,
at least that's not a consideration from now
## Further
We could refactor our package.json engines into only one using
`Yarn.set` etc
## Resource
- https://yarnpkg.com/configuration/yarnrc
- https://yarnpkg.com/features/constraints
## Note
- Not running constraints in `preInstall` hook as won't be effective on
fresh install
-
[engine-strict](https://docs.npmjs.com/cli/v8/using-npm/config#engine-strict)
is an npm-config
-
[devEngines](https://docs.npmjs.com/cli/v11/configuring-npm/package-json#devengines)
are npm feature too ( for instance pnpm current PR
https://github.com/pnpm/pnpm/issues/8153 )
## Conclusion
As always any suggestions are more than welcomed !
# Health Monitoring for Self-Hosted Instances
This PR implements basic health monitoring for self-hosted instances in
the admin panel.
## Service Status Checks
We're adding real-time health checks for:
- Redis Connection
- Database Connection
- Worker Status
- Message Sync Status
## Existing Functionality
We already have message sync and captcha counters that store aggregated
metrics in cache within a configurable time window (default: 5 minutes).
## New Endpoints
1. `/healthz` - Basic server health check for Kubernetes pod monitoring
2. `/healthz/{serviceName}` - Individual service health checks (returns
200 if healthy)
3. `/metricsz/{metricName}` - Time-windowed metrics (message sync,
captcha)
4. GraphQL resolver in admin panel for UI consumption
All endpoints use the same underlying service, with different
presentation layers for infrastructure and UI needs.
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
We are using useInView to detect if a row should be rendered or not.
# Behavior issue
When browsing an Index page (let's say People), then navigating to
another one (Company), then back to People, the rows were not displayed
anymore. For some reason the inView value was set to false in this case
# Fix
- I have updated the useInView (react-intersection-observer) package but
it did not fix it
- useInView provides a ref. However, I believe this was conflicting with
the draggableRef; cecause we are in a <table> we cannot add additional
containers and are forced to apply both refs to the <tr> (draggableRef +
inViewRef). I believe this was causing the issue. I have added the
inView ref to an empty <td> within the row which is achieving the same
goal without forcing to combine refs
- Do not render a source handle for the leaf nodes
- Upgrade the `@xyflow/react` library
| Before | After |
|--------|--------|
| 
| 
|
## Other options considered
React Flow exposes a hook to get the connections of the current node. I
tried to use this hook – which makes things way simpler – but I couldn't
find a way to make it work in Storybook. I had two options: 1. Set up
React Flow to render the nodes properly, 2. Mock the hook in Storybook.
The first option was hard to achieve as the `<Reactflow />` component
renders a whole flow, and it doesn't play well with the idea of
rendering a single node in a story.
The second option seemed overkill as mocking modules with Storybook is
not straightforward. See
https://storybook.js.org/docs/writing-stories/mocking-data-and-modules/mocking-modules.
I chose to keep the initial version of my code, written before I spot a
function simplifying the code. We can give it a look another time.
