## Context
We would like to start leveraging more messageFolders during messageSync
which would allow users to select folders they want to sync on their
mailbox. MessageFolder is an abstraction that means folder for
Microsoft, labels for Gmail, not supported for IMAP.
## What
1) We do not aim to build a FE now but we would like to take it into a
consideration if they have been modified through API. So **out of
scope**
2) MessageFolders will be synced regularly from Gmail, Microsoft. This
is currently partially done and improving it is **out of scope**
3) Change: we were having two synchronization mechanism so far:
FULL_SYNC (first time, or when no cursor is present) and PARTIAL_SYNC
(when we have a cursor, we can fetch the diff since the last pull). This
Full vs Partial mode was a high level concept. We now think it's an
driver implementation detail (and can be inferred from the existence of
a cursor). Why making this change now: as we are trying to pull several
folders, a folder could need a full sync if it had no cursor, and
another a partial if it has one. It does not make sense anymore to have
this full vs partial difference at MessageChannel level
4) Once we are sure this work, we can start syncing different folders on
Gmail side (the case for the user it to be able to fetch Promotion
label)
## Implementation strategy
1) Re-use PartialMessageList implementation / API and rename it to
MessageList at it's more complete
2) re-use driver level fullMessageList methods and call them
messageListWithoutCursor
3) make sure that these method are folder specific
## Tests
### Gmail
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
### Microsoft
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
### Imap
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
Fixes: https://github.com/twentyhq/twenty/issues/13110
I'm deprecating note.body and task.body to remove confusion between body
and bodyV2
What will be left but should be done later to avoid breaking changes:
- re-add a body field in the graphql API only that points to the same
bodyV2 field in SQL (need to be handled in fields and filter for note
and task)
- (wait some time)
- remove bodyV2 field
Context:
Users are complaining to see their workspace in a language they don't
know. This behavior is transient, happens on data model update and
disappear on refresh
I've check the cache for users that got the issue and did not spot any
weird language
==> I think we somehow fallback the the request header locale. I feel we
should always use the userWorkspace.locale, request locale should not be
used in BE in my opinion except for unauthenticated endpoints. I'm also
adding logs to understand the locale issue
In this PR:
rename user.workspaces into user.userWorkspaces which is more correct
improve / simplify LOCALES typing
Fixes https://github.com/twentyhq/twenty/issues/12867
Issue:
when you have a variable `toto` which is: `Record<string, MyType>` and
you do toto['xxx'], this will be typed as `MyType` instead of `MyType |
undefined`
Solutions:
- activate `noUncheckedIndexedAccess` check in tsconfig, this is the
preferred solution but will take time to get there (this raises 600+
errors)
- use a Map: cf https://github.com/twentyhq/twenty/pull/13125/files
- set the type to Partial<Record<string, MyType>>. Drawback is that when
you do Object.values(toto), you'll get `Array<MyType | undefined>`.
Hence why we have to filter these behind
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/d0a0bfed-c441-4e53-84c2-2da98ccbcf50"
/>
## Expected behavior
Described behavior regarding: (update | create) x (custom | standard) x
(icon, label, name, isSynced)
**Custom:**
- Field RELATION create: name, label, isSynced, icon should be editable
- Field RELATION update: name should not, icon label, isSynced should
- For other fields, icon, label, name, isSynced should be editable at
field creation | update
To simplify: Field RELATION name should not be editable at update
**Standards**
- Field: create does not makes sense
- Field: name should not, icon label, isSynced should (this will end up
in overrides)
To simplify, no Field RELATION edge case, name should not be editable at
update
**Note:** the FE logic is quite different as the UI is hiding some
details behind the syncWithLabel. See my comments and TODO there
## What I've tested:
(update | create) x (custom | standard) x (icon, label, name, isSynced,
description)
In this PR, I'm fixing a bug introduced in recent performance work on
the cache.
Bug context: https://github.com/twentyhq/twenty/issues/12865
Related PR opened by a contributor:
https://github.com/twentyhq/twenty/pull/13003
## Root cause
We cache all objectMetadataItems at graphql level : see
`useCachedMetadata` hook:
- instead of going through the regular resolvers, we direlcty load data
from the cache. However this data must be localized regarding labels and
descriptions
In a precedent refactoring, we introduced the notion of locale in the
cache key. However, the user locale was not properly taken into account
as we did not have the information in this hook.
## Fix
1. **Introduce locale in userWorkspace entity**. The locale is stored on
workspaceMember in each postgres workspaceSchema (workspace_xxx) which
is the alter ego of userWorkspace in postgres core schema. Note that we
can't store it in user as a user can be part of multiple workspaces (the
locale already there must be seen as a default for this user), and we
cannot rely on workspaceMember as we would need to query the
workspaceSchema in the authentication layer which we want to avoid for
performance reasons.
2. During request hydration from token (containing the userWorkspaceId),
we fetch the userWorkspace and store it in the Request (this impact both
AuthContext and Request interface)
3. Leverage userWorkspace.locale in the useCachedMetadata hook
## Additional notes
There is no need to change the way we store and retrieve the
object-metadata-maps object itself which is different from the graphql
layer cache. object-metadadata-maps are not localized
In this PR:
- add query hashKey to ObjectMetadataItems query graphql cache to avoid
caching outdated queries
- improve performance by removing ResolveField at FieldLevel and adding
this at resolver level
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
We were using a global ValidationPipe in main.ts. This is an issue as
@Controllers should return HttpExecption and @Resolvers should return
GraphqlErrors
Removing the global pipe and creating a ResolverValidationPipe able to
generate GraphqlError. We also need to handle the exception in a filter
to avoid nest to think it's unhandled and make it flow to logs
Next step:
- it would be nice to have both @UsePipes(ResolverValidationPipe) +
@UseFilters(GraphqlValidationExceptionFilter) come together. This should
be possible if we create a @GraphQLResolver annotation
## Context
Support button was missing for configuration having support enabled
(FrontApp)
<img width="1253" alt="image"
src="https://github.com/user-attachments/assets/930e3e0c-05a1-4a5b-820b-bb257f19fdde"
/>
## How
Recently, we changed some enums from lowercase to uppercase in graphql
## Problem resolution
supportDriver was typed as a string where we could have used
SupportDriver type. I'm exposing it in the graphql generated files to
re-use in the front so this issue cannot happen anymore
## Goal
We have identified that sync-metadata (which is called during new
workspace initialization) is slow mainly because of workspaceMigration
application (migration-runner module). This is due to the fact that we
use typeORM API to perform schema changes, which often query the
existing schema. As querying the existing schema is costly (especially
with ~1M existing columns) and as we already have what we need described
as metadata, we will use raw SQL directly. This should divide the
workspace initialization time by x2.
## How
This PR can be read in two commits:
1. Extract functions tied to column migrations in a separate service
(`workspace-migration-column.service`) + deprecate COMMENT column
migration type which is not useful since we are not using pg-graphql
anymore
2. Re-work `workspace-migration-column.service` to make it clearer + use
raw SQL
## Result
Before:
<img width="1367" alt="image"
src="https://github.com/user-attachments/assets/e730df7b-db7f-4433-9ce5-52841b010990"
/>
After:
<img width="1367" alt="image"
src="https://github.com/user-attachments/assets/72d2c2b1-2475-4541-a3d5-50b70824a2e4"
/>
## Manual Testing
- Sync-metadata OK
- Workspace init OK
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
For fresh install, we need the migrations to happen before the upgrade
command is triggered as the upgrade command is a NestJS command and the
app will try to load env variables from db
In this PR:
- deprecating listenClickOutside ComparePixel mode as this is not
accurate. We were using to avoid portal issue with CompareHtmlRef mode
but this is still an issue when portalled content overflows the
container.
- add ClickOutsideContext to specify excluded className so portal
children can use it easily (part of the tooling)
- fix stories
- remove avoidPortal from dropdown as this was not used
Several users have complained about not being able to read their emails
anymore.
This is because the find-messages post query hook is expecting
ObjectRecord[] as an input but is actually getting a graphql Connection
Typing was wrong. This PR fixes the typing and make sure the post query
hook always get an ObjectRecord[]
## What
This PR fixes a regression tied to the new relation format refactoring.
I'm also slightly improving the performance by decreasing the number of
queries.
## Considerations
1. I've started adding an integration test to cover relation creation
and deletion but we are still using the r`elation-metadata.service` to
create a relation and the `field-metadata.service` to delete it. As we
plan to fully deprecate the `relation-metadata.service`, I did not want
to invest into writing test tooling to change it in a few weeks so I've
reverted the test
2. We are still maintaining relationMetadata table up-to-date (so
deleting them when we delete the field). relationMetadata full
deprecation is up coming but I'm waiting a bit so we don't have breaking
changes in 0.53
This is a first PR to remove old relation logic
Next steps:
- remove relationMetadata from cache
- remove relationMetadata table content and structure
- refactor relationDefinition to leverage field.settings instead
## Context
While deploying the IS_NEW_RELATION_ENABLED (we don't compute relation
based on relationMetadata anymore) to existing workspace, I've tested to
run a sync-metadata post feature flag activation. This has raised two
issues:
- the workspaceMigration generator (which is over-complex and should be
refactored later) for fieldMetadata of type RELATION was not handling
settings update properly ;
- we need to delete existing fieldMetadata corresponding to the UUID
foreignKey as they are not needed anymore. This is handled as a 0.53
upgrade command as 0.53 will also come with the full removal of the old
relation system
---------
Co-authored-by: Etienne <45695613+etiennejouan@users.noreply.github.com>
Co-authored-by: prastoin <paul@twenty.com>
In this PR:
- this should fix the sync metadata for new relation system
This goes with the recent PR:
https://github.com/twentyhq/twenty/pull/11725
What we want:
- ONE_TO_MANY relations should have no joinColumn and no onDelete
- MANY_TO_ONE should have both
As per title:
- waitFor is a loop that waits for a condition to be filled, it should
be use to expect or in rare case to wait for element to be present in
the page (in most cases, you can use findByXXX)
- user actions should not be in this loop, otherwise they will be
triggered multiple times
In this PR:
- Remove SignUpLoading blank screen by an empty dark overlay =>
VerifyEffect
- Add ModalContent from pages themselves instead of using it the Layout.
This allow for empty dark overlay without showing an empty modal with
padding
1. Removing tokenPair internal variable of ApolloFactory. We will relay
on cookieStorage
2. setting the cookie explicitely instead of only relaying on recoil
cookieEffect which is too late
Remove FieldContext hotkey scope as:
- each Field should set its own hotkey scope (ex:
emails-field-input-{recordId} or emails-field-input for now)
- while opening a fieldInput, we should synchronously set the
corresponding hotkey scope
To cut this refactoring in half, I'm allowing all input to use
TableHotkeyScope.CellEditMode
## What
- Deprecate overlayscrollbars as we decided to follow the native
behavior
- rework on performances (avoid calling recoil states too much at field
level which is quite expensive)
- Also implements:
https://github.com/twentyhq/core-team-issues/issues/569
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
In this PR:
- allow to update settings on fields metadata (regression introduced by
a recent refactoring of fields-metadata update)
- revert changes introduced by
https://github.com/twentyhq/twenty/pull/11221
In this PR:
- Remove deactivated objects from ActivityTargetInlineCell record picker
- Prevent users to deactivate createdAt, updatedAt, deletedAt fields on
any objects
Still left:
- write unit tests on the assert utils
- write integration tests on field metadata service
- prevent users to deactivate createdAt, updatedAt, deletedAt on FE
In this PR, I'm:
- fixing the root cause (we should not try to render a RecordChip if the
record is not defined in RelationFromMany Display)
- fixing related typing issues
- we won't be able to catch the issue from TS perspective as
ObjectRecord is a Record of string, any
## What
This PR aims to make sure all application exceptions are captured
through react-error-boundaries
Once merged we will have:
- Root Level: AppErrorBoundary at the highest level (full screen) ==>
this one needs to be working in any case, not relying on Theme, was not
working
- Route Level: AppErrorBoundary in DefaultLayout (full screen) ==> this
was missing and it seems that error are not propagated outside of the
router, making errors triggered in CommandMenu or NavigationDrawer
missing
- Page Level: AppErrorBoundary in DefaultLayout write around the Page
itself (lower than CommandMenu + NavigationDrawer)
- Manually triggered: example in ClientConfigProvider
## Screenshots
App level (ex throw in IconsProvider)
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/18a14815-a203-4edf-b931-43068c3436ec"
/>
Route level (ex throw in CommandMenu)
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/ca066627-14c7-438e-a432-f0999a1f3b84"
/>
Page level (ex throw in RecordTable)
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/ffeaa935-02af-4762-8859-7a0ccf8b77e1"
/>
Manually Triggered (clientConfig, ex when backend is not up)
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/062d6d84-097a-4ed9-b6ce-763b8c27c659"
/>
## Context
Some users were able to set an empty URL as webhook targetUrl, which was
breaking the Webhook List and Detail pages
## Fix
- Making sure to protect getHostNameOrThrow by isValidUrl
- rework webhook form to prevent creation of invalid webhooks
Fixes https://github.com/twentyhq/twenty/issues/10822
The @scalar package we use to offer a REST api playground is quite heavy
(3k files compared to the 15k existing) and is not pre-build in the npm
package, which is not unusual.
This is increasing the memory need during vite build.
I'm increasing the RAM available for vite build.
Long term I recommend using a CDN here as this is not really a React
component so we won't benefit from any reactivity anyway. Exaclty like
we have done for FrontApp support chat integration
<img width="1058" alt="image"
src="https://github.com/user-attachments/assets/5412c6c1-7434-4b19-b9ac-e89f1cb614f3"
/>
Fixing:
- Export as PDF on empty note
- Command O (sub commands) not using the right contextStore
- BelongsToOne Field input triggering an error on open if no existing
relation record is pre-selected
In this PR, I'm specifying to vite build that
'@scalar/api-reference-react' is an external dependency and should not
be considered as a module we maintain (it won't get its own chunk at
build time and we won't generate sourcemaps on our end).
I'm not sure why vite is considering it internal in the first place (I
can see that it's generating .vue.js files, might be the first time we
are relying on a vue library)
Refactor to only have MultipleRecordPicker and SingleRecordPicker
What's done:
- SingleRecordPicker, MultipleRecordPicker
- RelationToOneInput
- RelationFromManyInput
- usage in TableCell, InlineCell, RelationDetailSection, Workflow
What's left:
- Make a pass on the app, to make sure the hotkeyScopes, clickOutside
are properly set
- Fix flashing on ActivityTarget
- add more tests on the code
Simplifying a lot the upgrade system.
New way to upgrade:
`yarn command:prod upgrade`
New way to write upgrade commands (all wrapping is done for you)
```
override async runOnWorkspace({
index,
total,
workspaceId,
options,
}: RunOnWorkspaceArgs): Promise<void> {}
```
Also cleaning CommandModule imports to make it lighter
While troubleshooting with a person self-hosting Twenty (v0.42.2), we
figured out that logs were missing on REST findMany, findOne and
duplicate endpoints
This PR fixes it
This PR is part 3 of RecordPicker refactoring.
It aims to remove all effects from:
- (low level layer) SingleRecordPicker, MultipleRecordPicker
- (higher level layer) RelationPicker, RelationToOneInput,
RelationFromManyInput, ActivityTargetInput...
In this PR, I'm re-grouping Effects in ActivityTarget section and
creating a hook that will be called on inlineCellOpen
## Context
We are experiencing a lot of re-rendering / flash on MultiRecordSelect /
SingleRecordSelect / RelationPicker.
This PR is a first step to refactor this components
## Scope
Only move files to uniformize and prepare for the upcoming refactoring
## Vision
- SingleRecordPicker
- MultipleRecordPicker
- sharing RecordPicker tooling (RecordPickerComponentInstanceContext,
searchState)
Used in other areas:
- RelationPicker (which is actually only a subcomponent of
RelationToOneFieldInput)
- RelationFromManyFieldInput
- WorkflowRelationFieldInput
- etc.
+ remove all effects
+ migrate to the latest APIs
+ make a pass on re-renders to remove them completely (we likely need to
use a bit more familyStates here)
Migrate and unify URL tooling in twenty-shared.
We now have:
- isValidHostname which follows our own business rules
- a zod schema that can be re-used in different context and leverages is
isValidHostname
- isValidUrl on top of the zod schema
- a getAbsoluteURl and getHostname on top of the zod schema
I have added a LOT of tests to cover all the cases I've found
Also fixes: https://github.com/twentyhq/twenty/issues/10147
In this PR:
- adding logs to track workspace creation performance
- refactor useIsWorkspaceSuspended to be more generic
- only fetch favorites and views if workspace is Active to avoid error
messages on sign up (workspace is not created yet)
Proposal:
- Add a method in ActiveWorkspaceCommand to loop over workspace safely
(add counter, add try / catch, provide datasource with fresh cache,
destroy datasource => as we do always do it)
Also in this PR:
- make sure we clear all dataSources (and not only the one on metadata
version in RAM)
Charles Bochet