## Expected behavior
Described behavior regarding: (update | create) x (custom | standard) x
(icon, label, name, isSynced)
**Custom:**
- Field RELATION create: name, label, isSynced, icon should be editable
- Field RELATION update: name should not, icon label, isSynced should
- For other fields, icon, label, name, isSynced should be editable at
field creation | update
To simplify: Field RELATION name should not be editable at update
**Standards**
- Field: create does not makes sense
- Field: name should not, icon label, isSynced should (this will end up
in overrides)
To simplify, no Field RELATION edge case, name should not be editable at
update
**Note:** the FE logic is quite different as the UI is hiding some
details behind the syncWithLabel. See my comments and TODO there
## What I've tested:
(update | create) x (custom | standard) x (icon, label, name, isSynced,
description)
In this PR, I'm fixing a bug introduced in recent performance work on
the cache.
Bug context: https://github.com/twentyhq/twenty/issues/12865
Related PR opened by a contributor:
https://github.com/twentyhq/twenty/pull/13003
## Root cause
We cache all objectMetadataItems at graphql level : see
`useCachedMetadata` hook:
- instead of going through the regular resolvers, we direlcty load data
from the cache. However this data must be localized regarding labels and
descriptions
In a precedent refactoring, we introduced the notion of locale in the
cache key. However, the user locale was not properly taken into account
as we did not have the information in this hook.
## Fix
1. **Introduce locale in userWorkspace entity**. The locale is stored on
workspaceMember in each postgres workspaceSchema (workspace_xxx) which
is the alter ego of userWorkspace in postgres core schema. Note that we
can't store it in user as a user can be part of multiple workspaces (the
locale already there must be seen as a default for this user), and we
cannot rely on workspaceMember as we would need to query the
workspaceSchema in the authentication layer which we want to avoid for
performance reasons.
2. During request hydration from token (containing the userWorkspaceId),
we fetch the userWorkspace and store it in the Request (this impact both
AuthContext and Request interface)
3. Leverage userWorkspace.locale in the useCachedMetadata hook
## Additional notes
There is no need to change the way we store and retrieve the
object-metadata-maps object itself which is different from the graphql
layer cache. object-metadadata-maps are not localized
In this PR:
- add query hashKey to ObjectMetadataItems query graphql cache to avoid
caching outdated queries
- improve performance by removing ResolveField at FieldLevel and adding
this at resolver level
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
We were using a global ValidationPipe in main.ts. This is an issue as
@Controllers should return HttpExecption and @Resolvers should return
GraphqlErrors
Removing the global pipe and creating a ResolverValidationPipe able to
generate GraphqlError. We also need to handle the exception in a filter
to avoid nest to think it's unhandled and make it flow to logs
Next step:
- it would be nice to have both @UsePipes(ResolverValidationPipe) +
@UseFilters(GraphqlValidationExceptionFilter) come together. This should
be possible if we create a @GraphQLResolver annotation
## Context
Support button was missing for configuration having support enabled
(FrontApp)
<img width="1253" alt="image"
src="https://github.com/user-attachments/assets/930e3e0c-05a1-4a5b-820b-bb257f19fdde"
/>
## How
Recently, we changed some enums from lowercase to uppercase in graphql
## Problem resolution
supportDriver was typed as a string where we could have used
SupportDriver type. I'm exposing it in the graphql generated files to
re-use in the front so this issue cannot happen anymore
## Goal
We have identified that sync-metadata (which is called during new
workspace initialization) is slow mainly because of workspaceMigration
application (migration-runner module). This is due to the fact that we
use typeORM API to perform schema changes, which often query the
existing schema. As querying the existing schema is costly (especially
with ~1M existing columns) and as we already have what we need described
as metadata, we will use raw SQL directly. This should divide the
workspace initialization time by x2.
## How
This PR can be read in two commits:
1. Extract functions tied to column migrations in a separate service
(`workspace-migration-column.service`) + deprecate COMMENT column
migration type which is not useful since we are not using pg-graphql
anymore
2. Re-work `workspace-migration-column.service` to make it clearer + use
raw SQL
## Result
Before:
<img width="1367" alt="image"
src="https://github.com/user-attachments/assets/e730df7b-db7f-4433-9ce5-52841b010990"
/>
After:
<img width="1367" alt="image"
src="https://github.com/user-attachments/assets/72d2c2b1-2475-4541-a3d5-50b70824a2e4"
/>
## Manual Testing
- Sync-metadata OK
- Workspace init OK
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
For fresh install, we need the migrations to happen before the upgrade
command is triggered as the upgrade command is a NestJS command and the
app will try to load env variables from db
In this PR:
- deprecating listenClickOutside ComparePixel mode as this is not
accurate. We were using to avoid portal issue with CompareHtmlRef mode
but this is still an issue when portalled content overflows the
container.
- add ClickOutsideContext to specify excluded className so portal
children can use it easily (part of the tooling)
- fix stories
- remove avoidPortal from dropdown as this was not used
Several users have complained about not being able to read their emails
anymore.
This is because the find-messages post query hook is expecting
ObjectRecord[] as an input but is actually getting a graphql Connection
Typing was wrong. This PR fixes the typing and make sure the post query
hook always get an ObjectRecord[]
## What
This PR fixes a regression tied to the new relation format refactoring.
I'm also slightly improving the performance by decreasing the number of
queries.
## Considerations
1. I've started adding an integration test to cover relation creation
and deletion but we are still using the r`elation-metadata.service` to
create a relation and the `field-metadata.service` to delete it. As we
plan to fully deprecate the `relation-metadata.service`, I did not want
to invest into writing test tooling to change it in a few weeks so I've
reverted the test
2. We are still maintaining relationMetadata table up-to-date (so
deleting them when we delete the field). relationMetadata full
deprecation is up coming but I'm waiting a bit so we don't have breaking
changes in 0.53
This is a first PR to remove old relation logic
Next steps:
- remove relationMetadata from cache
- remove relationMetadata table content and structure
- refactor relationDefinition to leverage field.settings instead
## Context
While deploying the IS_NEW_RELATION_ENABLED (we don't compute relation
based on relationMetadata anymore) to existing workspace, I've tested to
run a sync-metadata post feature flag activation. This has raised two
issues:
- the workspaceMigration generator (which is over-complex and should be
refactored later) for fieldMetadata of type RELATION was not handling
settings update properly ;
- we need to delete existing fieldMetadata corresponding to the UUID
foreignKey as they are not needed anymore. This is handled as a 0.53
upgrade command as 0.53 will also come with the full removal of the old
relation system
---------
Co-authored-by: Etienne <45695613+etiennejouan@users.noreply.github.com>
Co-authored-by: prastoin <paul@twenty.com>
In this PR:
- this should fix the sync metadata for new relation system
This goes with the recent PR:
https://github.com/twentyhq/twenty/pull/11725
What we want:
- ONE_TO_MANY relations should have no joinColumn and no onDelete
- MANY_TO_ONE should have both
As per title:
- waitFor is a loop that waits for a condition to be filled, it should
be use to expect or in rare case to wait for element to be present in
the page (in most cases, you can use findByXXX)
- user actions should not be in this loop, otherwise they will be
triggered multiple times
In this PR:
- Remove SignUpLoading blank screen by an empty dark overlay =>
VerifyEffect
- Add ModalContent from pages themselves instead of using it the Layout.
This allow for empty dark overlay without showing an empty modal with
padding
1. Removing tokenPair internal variable of ApolloFactory. We will relay
on cookieStorage
2. setting the cookie explicitely instead of only relaying on recoil
cookieEffect which is too late
Remove FieldContext hotkey scope as:
- each Field should set its own hotkey scope (ex:
emails-field-input-{recordId} or emails-field-input for now)
- while opening a fieldInput, we should synchronously set the
corresponding hotkey scope
To cut this refactoring in half, I'm allowing all input to use
TableHotkeyScope.CellEditMode
## What
- Deprecate overlayscrollbars as we decided to follow the native
behavior
- rework on performances (avoid calling recoil states too much at field
level which is quite expensive)
- Also implements:
https://github.com/twentyhq/core-team-issues/issues/569
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
In this PR:
- allow to update settings on fields metadata (regression introduced by
a recent refactoring of fields-metadata update)
- revert changes introduced by
https://github.com/twentyhq/twenty/pull/11221
In this PR:
- Remove deactivated objects from ActivityTargetInlineCell record picker
- Prevent users to deactivate createdAt, updatedAt, deletedAt fields on
any objects
Still left:
- write unit tests on the assert utils
- write integration tests on field metadata service
- prevent users to deactivate createdAt, updatedAt, deletedAt on FE
In this PR, I'm:
- fixing the root cause (we should not try to render a RecordChip if the
record is not defined in RelationFromMany Display)
- fixing related typing issues
- we won't be able to catch the issue from TS perspective as
ObjectRecord is a Record of string, any
## What
This PR aims to make sure all application exceptions are captured
through react-error-boundaries
Once merged we will have:
- Root Level: AppErrorBoundary at the highest level (full screen) ==>
this one needs to be working in any case, not relying on Theme, was not
working
- Route Level: AppErrorBoundary in DefaultLayout (full screen) ==> this
was missing and it seems that error are not propagated outside of the
router, making errors triggered in CommandMenu or NavigationDrawer
missing
- Page Level: AppErrorBoundary in DefaultLayout write around the Page
itself (lower than CommandMenu + NavigationDrawer)
- Manually triggered: example in ClientConfigProvider
## Screenshots
App level (ex throw in IconsProvider)
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/18a14815-a203-4edf-b931-43068c3436ec"
/>
Route level (ex throw in CommandMenu)
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/ca066627-14c7-438e-a432-f0999a1f3b84"
/>
Page level (ex throw in RecordTable)
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/ffeaa935-02af-4762-8859-7a0ccf8b77e1"
/>
Manually Triggered (clientConfig, ex when backend is not up)
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/062d6d84-097a-4ed9-b6ce-763b8c27c659"
/>
## Context
Some users were able to set an empty URL as webhook targetUrl, which was
breaking the Webhook List and Detail pages
## Fix
- Making sure to protect getHostNameOrThrow by isValidUrl
- rework webhook form to prevent creation of invalid webhooks
Fixes https://github.com/twentyhq/twenty/issues/10822
The @scalar package we use to offer a REST api playground is quite heavy
(3k files compared to the 15k existing) and is not pre-build in the npm
package, which is not unusual.
This is increasing the memory need during vite build.
I'm increasing the RAM available for vite build.
Long term I recommend using a CDN here as this is not really a React
component so we won't benefit from any reactivity anyway. Exaclty like
we have done for FrontApp support chat integration
<img width="1058" alt="image"
src="https://github.com/user-attachments/assets/5412c6c1-7434-4b19-b9ac-e89f1cb614f3"
/>
Fixing:
- Export as PDF on empty note
- Command O (sub commands) not using the right contextStore
- BelongsToOne Field input triggering an error on open if no existing
relation record is pre-selected
In this PR, I'm specifying to vite build that
'@scalar/api-reference-react' is an external dependency and should not
be considered as a module we maintain (it won't get its own chunk at
build time and we won't generate sourcemaps on our end).
I'm not sure why vite is considering it internal in the first place (I
can see that it's generating .vue.js files, might be the first time we
are relying on a vue library)
Refactor to only have MultipleRecordPicker and SingleRecordPicker
What's done:
- SingleRecordPicker, MultipleRecordPicker
- RelationToOneInput
- RelationFromManyInput
- usage in TableCell, InlineCell, RelationDetailSection, Workflow
What's left:
- Make a pass on the app, to make sure the hotkeyScopes, clickOutside
are properly set
- Fix flashing on ActivityTarget
- add more tests on the code
Simplifying a lot the upgrade system.
New way to upgrade:
`yarn command:prod upgrade`
New way to write upgrade commands (all wrapping is done for you)
```
override async runOnWorkspace({
index,
total,
workspaceId,
options,
}: RunOnWorkspaceArgs): Promise<void> {}
```
Also cleaning CommandModule imports to make it lighter
While troubleshooting with a person self-hosting Twenty (v0.42.2), we
figured out that logs were missing on REST findMany, findOne and
duplicate endpoints
This PR fixes it
This PR is part 3 of RecordPicker refactoring.
It aims to remove all effects from:
- (low level layer) SingleRecordPicker, MultipleRecordPicker
- (higher level layer) RelationPicker, RelationToOneInput,
RelationFromManyInput, ActivityTargetInput...
In this PR, I'm re-grouping Effects in ActivityTarget section and
creating a hook that will be called on inlineCellOpen
## Context
We are experiencing a lot of re-rendering / flash on MultiRecordSelect /
SingleRecordSelect / RelationPicker.
This PR is a first step to refactor this components
## Scope
Only move files to uniformize and prepare for the upcoming refactoring
## Vision
- SingleRecordPicker
- MultipleRecordPicker
- sharing RecordPicker tooling (RecordPickerComponentInstanceContext,
searchState)
Used in other areas:
- RelationPicker (which is actually only a subcomponent of
RelationToOneFieldInput)
- RelationFromManyFieldInput
- WorkflowRelationFieldInput
- etc.
+ remove all effects
+ migrate to the latest APIs
+ make a pass on re-renders to remove them completely (we likely need to
use a bit more familyStates here)
Migrate and unify URL tooling in twenty-shared.
We now have:
- isValidHostname which follows our own business rules
- a zod schema that can be re-used in different context and leverages is
isValidHostname
- isValidUrl on top of the zod schema
- a getAbsoluteURl and getHostname on top of the zod schema
I have added a LOT of tests to cover all the cases I've found
Also fixes: https://github.com/twentyhq/twenty/issues/10147
In this PR:
- adding logs to track workspace creation performance
- refactor useIsWorkspaceSuspended to be more generic
- only fetch favorites and views if workspace is Active to avoid error
messages on sign up (workspace is not created yet)
Proposal:
- Add a method in ActiveWorkspaceCommand to loop over workspace safely
(add counter, add try / catch, provide datasource with fresh cache,
destroy datasource => as we do always do it)
Also in this PR:
- make sure we clear all dataSources (and not only the one on metadata
version in RAM)
In this huge (sorry!) PR:
- introducing objectMetadataItem in contextStore instead of
objectMetadataId which is more convenient
- splitting some big hooks into smaller parts to avoid re-renders
- removing Effects to avoid re-renders (especially onViewChange)
- making the view prefetch separate from favorites to avoid re-renders
- making the view prefetch load a state and add selectors on top of it
to avoir re-renders
As a result, the performance is WAY better (I suspect the favorite
implementation to trigger a lot of re-renders unfortunately).
However, we are still facing a random app freeze on view creation. I
could not investigate the root cause. As this seems to be already there
in the precedent release, we can move forward but this seems a urgent
follow up to me ==> EDIT: I've found the root cause after a few ours of
deep dive... an infinite loop in RecordTableNoRecordGroupBodyEffect...
prastoin edit: close https://github.com/twentyhq/twenty/issues/10253
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
Co-authored-by: prastoin <paul@twenty.com>
In this PR, I'm simplifying the lastVisitedView / Object logic:
- removing fallback logic as it's not useful
- splitting hooks into smaller hooks (to avoir re-renders)
- removing componentState on those states that are global
While making sure the upgrade-0.42 command was working as expected to
upgrade from 0.41 to 0.42, we've detected that the standardId of the new
bodyV2 (type RICH_TEXT_V2) was not properly set ; standardId was not
correct.
This was forcing the sync-metadata command to try to re-create the field
again.
We are using useInView to detect if a row should be rendered or not.
# Behavior issue
When browsing an Index page (let's say People), then navigating to
another one (Company), then back to People, the rows were not displayed
anymore. For some reason the inView value was set to false in this case
# Fix
- I have updated the useInView (react-intersection-observer) package but
it did not fix it
- useInView provides a ref. However, I believe this was conflicting with
the draggableRef; cecause we are in a <table> we cannot add additional
containers and are forced to apply both refs to the <tr> (draggableRef +
inViewRef). I believe this was causing the issue. I have added the
inView ref to an empty <td> within the row which is achieving the same
goal without forcing to combine refs
Charles Bochet