In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
# Introduction
Big diff a lot of tests and snapshots ( real diff < 500+ )
close https://github.com/twentyhq/twenty/issues/12117
close https://github.com/twentyhq/twenty/issues/12133
## What has been done here
Implemented a strong integration coverage on both fieldmetadata`SELECT`
`UPDATE` and `CREATE`.
Implemented server side validation for the options `value` `label` `id`
and collision issue with also `position`
We could improve:
- Position validation
- DefaultValue validation
## Update
```ts
PASS test/integration/metadata/suites/field-metadata/update-one-field-metadata-select.integration-spec.ts (41.054 s)
Field metadata select update tests group
✓ Update should succeed with provided option id (2565 ms)
✓ Update should succeed with valid default value (1469 ms)
✓ Update should succeed with various options id (1257 ms)
✓ Update should succeed without option id (1286 ms)
✓ Update should trim option values (1366 ms)
✓ Update should succeed with default value and no options (1122 ms)
✓ Update should fail with unknown default value and no options (1075 ms)
✓ Update should fail with only white spaces id (1195 ms)
✓ Update should fail with empty string id (1058 ms)
✓ Update should fail with null id (1066 ms)
✓ Update should fail with not a string id (1098 ms)
✓ Update should fail with too long id (1373 ms)
✓ Update should fail with only white spaces label (1034 ms)
✓ Update should fail with empty string label (1057 ms)
✓ Update should fail with null label (1100 ms)
✓ Update should fail with not a string label (1144 ms)
✓ Update should fail with too long label (1273 ms)
✓ Update should fail with only white spaces value (1385 ms)
✓ Update should fail with empty string value (1035 ms)
✓ Update should fail with null value (1068 ms)
✓ Update should fail with not a string value (1021 ms)
✓ Update should fail with too long value (1134 ms)
✓ Update should fail with invalid option id (1137 ms)
✓ Update should fail with empty options (1238 ms)
✓ Update should fail with invalid option value format (1104 ms)
✓ Update should fail with comma in option label (1004 ms)
✓ Update should fail with duplicated option values (1015 ms)
✓ Update should fail with duplicated option ids (1079 ms)
✓ Update should fail with duplicated option positions (1266 ms)
✓ Update should fail with duplicated trimmed option values (1220 ms)
✓ Update should fail with undefined option label (1029 ms)
✓ Update should fail with an invalid default value (1142 ms)
✓ Update should fail with an unknown default value (1081 ms)
✓ Update should fail with undefined option value (1086 ms)
Test Suites: 1 passed, 1 total
Tests: 34 passed, 34 total
Snapshots: 28 passed, 28 total
Time: 41.079 s
```
## Create
```ts
PASS test/integration/metadata/suites/field-metadata/create-one-field-metadata-select.integration-spec.ts (38.292 s)
Field metadata select creation tests group
✓ Create should succeed with provided option id (2096 ms)
✓ Create should succeed with valid default value (1316 ms)
✓ Create should succeed with various options id (1113 ms)
✓ Create should succeed without option id (1378 ms)
✓ Create should trim option values (1296 ms)
✓ Create should fail with only white spaces id (1000 ms)
✓ Create should fail with empty string id (1325 ms)
✓ Create should fail with null id (1060 ms)
✓ Create should fail with not a string id (1142 ms)
✓ Create should fail with too long id (1321 ms)
✓ Create should fail with only white spaces label (999 ms)
✓ Create should fail with empty string label (1163 ms)
✓ Create should fail with null label (1198 ms)
✓ Create should fail with not a string label (1678 ms)
✓ Create should fail with too long label (1527 ms)
✓ Create should fail with only white spaces value (1200 ms)
✓ Create should fail with empty string value (1102 ms)
✓ Create should fail with null value (1037 ms)
✓ Create should fail with not a string value (1462 ms)
✓ Create should fail with too long value (896 ms)
✓ Create should fail with invalid option id (997 ms)
✓ Create should fail with empty options (1058 ms)
✓ Create should fail with invalid option value format (1190 ms)
✓ Create should fail with comma in option label (1142 ms)
✓ Create should fail with duplicated option values (872 ms)
✓ Create should fail with duplicated option ids (860 ms)
✓ Create should fail with duplicated option positions (1002 ms)
✓ Create should fail with duplicated trimmed option values (1336 ms)
✓ Create should fail with undefined option label (754 ms)
✓ Create should fail with an invalid default value (696 ms)
✓ Create should fail with an unknown default value (678 ms)
✓ Create should fail with undefined option value (699 ms)
✓ Create should fail with null options (720 ms)
✓ Create should fail with undefined options (686 ms)
Test Suites: 1 passed, 1 total
Tests: 34 passed, 34 total
Snapshots: 29 passed, 29 total
Time: 38.314 s
```
## Conclusion
As always any suggestions are welcomed ! Please let me know
## Discussion about validation governance
### Front
Front side will be dealing with zod validations schema that he will
handle and maintain by himself
### Back validation instances
- Validation hold through DTO declarations ( run by yoga through the
resolvers )
- Server programmatic validation and exceptions handling ( run through
the services )
For this refactor/fix we decided to stick to the current implementation
only touching the `Server programmatic validation and exceptions
handling` we will handle validation centralization when we will onboard
the `nestjs-query` deprecation/integration refactor.
### Vision
In the best of the world we could think of an intermediary model that
will handle and take responsibility of the validation decorators that
would be run programmatically through the service, Yoga would still
consume it ? then we would need to have enough grain in the service to
know the input has already validated
## Notes
Introduced zod back side in order to handle very atomic and primitive
validation
This is a first PR to remove old relation logic
Next steps:
- remove relationMetadata from cache
- remove relationMetadata table content and structure
- refactor relationDefinition to leverage field.settings instead
Fixes#10793
This PR is a work in progress.
**Still left to fix:**
- [x] When disabling synchronization of labels / api names, the edited
labels should be set to the English version. Currently the client just
send the localized versions together with the `isLabelSyncedWithName`
change. Could be an easy fix.
- [ ] Sometimes flipping the switch don't trigger the update function,
may be a regression as it seems to affect the custom objects too.
- [ ] There is a frontend problem where the labels inputs don't reflect
the changes made. When enabling back synchronisation after editing
labels, they are correctly back to their base values (backend,
navigation breadcrumb, etc) but the label inputs still have the old
values (switching pages will put them back to normal). I suspect this
could be linked to the above problem.
- [ ] API names are still displayed for standard objects per (kept them
for debugging, trivial fix)
- [ ] `SettingsDataModelObjectAboutForm` have a `disableEdition`
parameter which is now used only for a few fields, not sure if it's
worth keeping because it's a bit misleading since it doesn't "disable"
much?
- [ ] I don't know what these do, but I have seen "Remote" object types.
Not sure if they work with my patch or not (I don't know how to test
them)
- [ ] Make it work with metadata synchronisation
**What should work:**
- Disabling synchronization of standard objects should work, label
inputs should no longer be disabled
- Modifying labels should work
- Enabling back synchronization should reset back the labels to the base
value and disable the label inputs again (minus the mentioned display
bug)
- The synchronisation switch should still work as expected for custom
objects
- Creating custom objects should still work (it uses the same form)
---------
Signed-off-by: AFCMS <afcm.contact@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Adding SettingsPermissionsGuard to execute permission check.
The guard is added directly in resolver, either at resolver level (ex:
roles) or resolver-endpoint level (ex: metadata). this can be challenged
!
`No Value` view groups wasn't properly created when we select a group by
field metadata, this PR fix the issue.
Also a script is added to backfill the current view groups.
---------
Co-authored-by: Marie <51697796+ijreilly@users.noreply.github.com>
In this PR:
- removing ugprade-0.24 commands as we are releasing 0.30
- introducing cache:flush command
- refactoring upgrade command and sync-metadata command to use the
ActiveWorkspacesCommand so they consistently run on all workspaces or
selected workspaces
Fixes:
- clear localStorage on sign out
- fix missing workspaceMember in verify resolver
- do not throw on datasource already destroyed exception which can
happen with race condition when several resolvers are resolving in
parallel
In this PR:
1. Refactor guards to avoid duplicated queries: WorkspaceAuthGuard and
UserAuthGuard only check for existence of workspace and user in the
request without querying the database
## Context
The goal is to replace pg_graphql with our own ORM wrapper (TwentyORM).
This PR tries to add some parsing logic to convert graphql requests to
send to the ORM to replace pg_graphql implementation.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Fix https://github.com/twentyhq/twenty/issues/6669
- create a commun function `startWorkflowRun` that both create the run
object and the job for executing the workflow
- use it in both the `workflowEventJob` and the `runWorkflowVersion`
endpoint
Bonus:
- use filtering for exceptions instead of a util. It avoids doing a try
catch in all endpoint
## Context
As we grow, the messaging scripts are experiencing performance issues
forcing us to temporarily disable them on the cloud.
While investigating the performance, I have noticed that generating the
entity schema (for twentyORM) in the repository is taking ~500ms locally
on my Mac M2 so likely more on pods. Caching the entitySchema then!
I'm also clarifying naming around schemaVersion and cacheVersions ==>
both are renamed workspaceMetadataVersion and migrated to the workspace
table (the workspaceCacheVersion table is dropped).
This pull request introduces a new `FieldMetadataType` called `ACTOR`.
The primary objective of this new type is to add an extra column to the
following objects: `person`, `company`, `opportunity`, `note`, `task`,
and all custom objects.
This composite type contains three properties:
- `source`
```typescript
export enum FieldActorSource {
EMAIL = 'EMAIL',
CALENDAR = 'CALENDAR',
API = 'API',
IMPORT = 'IMPORT',
MANUAL = 'MANUAL',
}
```
- `workspaceMemberId`
- This property can be `undefined` in some cases and refers to the
member who created the record.
- `name`
- Serves as a fallback if the `workspaceMember` is deleted and is used
for other source types like `API`.
### Functionality
The pre-hook system has been updated to allow real-time argument
updates. When a record is created, a pre-hook can now compute and update
the arguments accordingly. This enhancement enables the `createdBy`
field to be populated with the correct values based on the
`authContext`.
The `authContext` now includes:
- An optional User entity
- An optional ApiKey entity
- The workspace entity
This provides access to the necessary data for the `createdBy` field.
In the GraphQL API, only the `source` can be specified in the
`createdBy` input. This allows the front-end to specify the source when
creating records from a CSV file.
### Front-End Handling
On the front-end, `orderBy` and `filter` are only applied to the name
property of the `ACTOR` composite type. Currently, we are unable to
apply these operations to the workspace member relation. This means that
if a workspace member changes their first name or last name, there may
be a mismatch because the name will differ from the new one. The name
displayed on the screen is based on the workspace member entity when
available.
### Missing Components
Currently, this PR does not include a `createdBy` value for the `MAIL`
and `CALENDAR` sources. These records are created in a job, and at
present, we only have access to the workspaceId within the job. To
address this, we should use a function similar to
`loadServiceWithContext`, which was recently removed from `TwentyORM`.
This function would allow us to pass the `authContext` to the jobs
without disrupting existing jobs.
Another PR will be created to handle these cases.
### Related Issues
Fixes issue #5155.
### Additional Notes
This PR doesn't include the migrations of the current records and views.
Everything works properly when the database is reset but this part is
still missing for now. We'll add that in another PR.
- There is a minor issue: front-end tests are broken since this commit:
[80c0fc7ff1).
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
Services exceptions are not catch when the endpoint comes from an
auto-resolver.
We want to remove auto-resolver but it requires to implement pagination
by ourselves.
As a quick fix, here are interceptors that will trigger the exception
handler.
I had a hard time making it generic so I finally added one interceptor
for each since this is not supposed to stay
Fixes#5276.
Updates were not triggering a cache version incrementation because they
do not trigger migrations while that is where the caching version logic
was.
We have decided to move the cache incrementation logic to the services.
New strategy:
- add settings field on FieldMetadata. Contains a boolean isIdField and
for numbers, a precision
- if idField, the graphql scalar returned will be a GraphQL id. This
will allow the app to work even for ids that are not uuid
- remove globals dateScalar and numberScalar modes. These were not used
- set limit as Integer
- check manually in query runner mutations that we send a valid id
Todo left:
- remove WorkspaceBuildSchemaOptions since this is not used anymore.
Will do in another PR
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
Co-authored-by: Weiko <corentin@twenty.com>
**Context**
cf. feature request
[#4597](https://github.com/twentyhq/twenty/issues/4597)
Enables deletion of custom fields that aren't active nor of type
relation
Also
1. renamed a misnamed file
2. deleted redundant hook BeforeDeleteOneField as it seemed best to move
the logic to the resolver instead
**How was it tested?**
Did not write unit tests as code is to be migrated (discussed with
@Weiko).
Locally tested.
---------
Co-authored-by: Marie Stoppa <mariestoppa@MacBook-Pro-de-Marie.local>
