## Context
- Removing search* integration tests instead of fixing them because they
will be replaced by global search very soon
- Fixed billing + add missing seeds to make them work
- Fixed integration tests not using consistently the correct "test" db
- Fixed ci not running the with-db-reset configuration due to nx
configuration being used twice for different level of the command
- Enriched .env.test
- Fixed parts where exceptions were not thrown properly and not caught
by exception handler to convert to 400 when needed
- Refactored feature flag service that had 2 different implementations
in lab and admin panel + added tests
- Fixed race condition when migrations are created at the same timestamp
and doing the same type of operation, in this case object deletion could
break because table could be deleted earlier than its relations
- Fixed many integration tests that were not up to date since the CI has
been broken for a while
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
## Context
Workspace creation and more specifically sync-metadata performances are
bad at the moment. We are trying to identify bottlenecks and one of the
root causes is the migration runner that can take up to 10s when setting
up a new workspaces with all its tables.
First observation is we do a lot of things sequentially, mostly to make
the code easier to read and debug but it impacts performances. For
example, a table creation is done in two steps, we first ask typeorm to
create the table then ask typeorm to create columns (and sometimes
columns one by one), each instruction can take time because typeorm
seems to do some checks internally.
The proposition here is to try to merge migrations when possible, for
example when we create a table we want the migration to also contain the
columns it will contain so we can ask typeorm to add the columns at the
same time. We are also using batch operations when possible (addColumns
instead of addColumn, dropColumns instead of dropColumn)
Still, we could go further with foreign keys creations or/and try with
raw query directly.
## Test
New workspace creation:
See screenshot, 9865.40233296156ms is on main, the rest is after the
changes:
<img width="610" alt="Screenshot 2025-02-28 at 09 27 21"
src="https://github.com/user-attachments/assets/42e880ff-279e-4170-b705-009e4b72045c"
/>
ResetDB and Sync-metadata on an existing workspace commands still work
Fixes https://github.com/twentyhq/twenty/issues/8810
Fixes https://github.com/twentyhq/twenty/issues/5268
Fixes https://github.com/twentyhq/twenty/issues/8971
- Fixing Task/Note creation not sending position during creation
- Adding a command to backfill position being null, using existing
backfill command.
- Removed unused backfill job.
- Updated workspace entities to set position non-nullable and set a
default value to make it non-required on the API
- Updated position factory to set a default position for all objects
having a POSITION field instead of only company/people
- Moved the try/catch in each resolver factory calling
GraphqlQueryRunnerException handler, makes more sense to call it in the
actual graphql-query-runner and removing some duplicate codes
- Adding validations for input in QueryRunnerArgs factories
- Allow sync-metadata to override and sync defaultValues for certain
field types (that can't be updated by users)
- Removing health-check from sync-metadata command during force mode to
improve performances
## Context
The recent addition of object renaming introduced issues with enum
names. Enum names should follow the pattern
`${schemaName}.${tableName}_${columnName}_enum`. To address this, and to
allow users to customize the API name (which is included in the enum
name, columnName), this PR implements behavior similar to object
renaming by introducing a `isLabelSyncedWithName` boolean.
<img width="624" alt="Screenshot 2024-12-02 at 11 58 49"
src="https://github.com/user-attachments/assets/690fb71c-83f0-4922-80c0-946c92dacc30">
<img width="596" alt="Screenshot 2024-12-02 at 11 58 39"
src="https://github.com/user-attachments/assets/af9a0037-7cf5-40c3-9ed5-d51b340c8087">
## Context
Enum are named after this pattern
`${schema}_${tableName}_${columnName}_enum` however now that we allowed
table name update, we need to make sure their enums are renamed as well.
## Issue
Some users were facing issues while updating SELECT or MULTISELECT
options:
```
Error executing migration QueryFailedError: column "undefined" does not exist
at PostgresQueryRunner.query (/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async WorkspaceMigrationEnumService.migrateEnumValues (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/services/workspace-migration-enum.service.js:101:13)
at async WorkspaceMigrationEnumService.alterEnum (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/services/workspace-migration-enum.service.js:54:9)
at async WorkspaceMigrationRunnerService.alterColumn (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:254:13)
at async WorkspaceMigrationRunnerService.handleColumnChanges (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:202:21)
at async WorkspaceMigrationRunnerService.handleTableChanges (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:94:25)
at async WorkspaceMigrationRunnerService.executeMigrationFromPendingMigrations (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:60:17)
at async WorkspaceExecutePendingMigrationsCommand.run (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/commands/workspace-execute-pending-migrations.command.js:24:9)
at async Command.<anonymous> (/app/node_modules/nest-commander/src/command-runner.service.js:162:24) {
query: '\n' +
' UPDATE "workspace_7i7bwawp7keh3gel1g69jropc"."_funnelInProductsRu"\n' +
' SET "reasonForStageTransition" = undefined\n' +
" WHERE id='2af8db61-5f75-46de-8b1a-97e312937e06'\n" +
' ',
parameters: undefined,
driverError: error: column "undefined" does not exist
```
## Root cause
Internally, while migrating enum values, we are doing:
```
const values = await queryRunner.query(
`SELECT id, "${oldColumnName}" FROM "${schemaName}"."${tableName}"`,
);
let val = value[oldColumnName];
```
oldColumnName being: `${columnDefinition.columnName}_old_${v4()}`;
However, TypeORM only supports 63 bits identifiers:
https://github.com/typeorm/typeorm/issues/3118, but silently truncate
the identifer so if oldColumnName gets larger than 63 characters,
`value[oldColumnName]` is undefined.
## Fix
We only need a temporary and unique temporary name, no need to take into
account the previous columnName here
## Context
- Fixing folder structure where 0.33 was inside 0.32.
- Updating the command so it runs enforce uniqueness on all tables
- Updating workspaces entities so the sync metadata adds the index
- Removing unique constraint creation in 0.32 as we have a dependency on
sync-metadata and sync-metadata has a dependency on it if we keep it.
- sync-metadata does not support well the deletion of activity +
activityTarget, silently swallow the exception for now
`isUnique` was passed to TypeORM's column creation API resulting in
double index creation because it's already done via the decorator and
then in `WorkspaceMigrationIndexFactory`
It would be interesting to move it at this field level in a later step,
which is why I also fixed `CompositeColumnActionFactory` to pass
isUnique on the correct columns, even though it's being ignored later on
By default, when custom fields are created, a searchVector field is
created based on the "name" field, which is also the label identifier by
default.
When this label identifier is updated, we want to update the
searchVector field to use this field as searchable field instead, if it
is of "searchable type" (today it is only possible to select a text or
number field as label identifier, while number fields are not
searchable).
## Context
This PR removes workspace-query-runner/builder in preparation for fully
deprecating pg_graphql
next steps: Remove from the setup and make a command to remove comments
on schema/tables related to pg_graphql
Add support for indexes on composite fields and unicity constraint on
indexes
This pull request includes several changes across multiple files to
improve error handling, enforce unique constraints, and update database
migrations. The most important changes include updating error messages
for snack bars, adding a new command to enforce unique constraints, and
updating database migrations to include new fields and constraints.
### Error Handling Improvements:
*
[`packages/twenty-front/src/modules/error-handler/components/PromiseRejectionEffect.tsx`](diffhunk://#diff-e7dc05ced8e4730430f5c7fcd0c75b3aa723da438c26e0bef8130b614427dd9aL23-R23):
Updated error messages in `enqueueSnackBar` to use `error.message`
directly.
*
[`packages/twenty-front/src/modules/object-metadata/hooks/useFindManyObjectMetadataItems.ts`](diffhunk://#diff-74c126d6bc7a5ed6b63be994d298df6669058034bfbc367b11045f9f31a3abe6L44-R46):
Simplified error messages in `enqueueSnackBar`.
*
[`packages/twenty-front/src/modules/object-record/hooks/useFindDuplicateRecords.ts`](diffhunk://#diff-af23a1d99639a66c251f87473e63e2b7bceaa4ee4f70fedfa0fcffe5c7d79181L56-R58):
Simplified error messages in `enqueueSnackBar`.
*
[`packages/twenty-front/src/modules/object-record/hooks/useHandleFindManyRecordsError.ts`](diffhunk://#diff-da04296cbe280202a1eaf6b1244a30490d4f400411bee139651172c59719088eL22-R24):
Simplified error messages in `enqueueSnackBar`.
### New Command for Unique Constraints:
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-enforce-unique-constraints.command.ts`](diffhunk://#diff-8337096c8c80dd2619a5ba691ae5145101f8ae0368a75192a050047e8c6ab7cbR1-R159):
Added a new command to enforce unique constraints on company domain
names and person emails.
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-upgrade-version.command.ts`](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR13-R14):
Integrated the new `EnforceUniqueConstraintsCommand` into the upgrade
process.
[[1]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR13-R14)
[[2]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR31)
[[3]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR64-R68)
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-upgrade-version.module.ts`](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R7):
Registered the new `EnforceUniqueConstraintsCommand` in the module.
[[1]](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R7)
[[2]](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R24)
### Database Migrations:
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726757368824-migrationDebt.ts`](diffhunk://#diff-c450aeae7bc0ef4416a0ade2dc613ca3f688629f35d2a32f90a09c3f494febdcR1-R53):
Added a migration to update the `relationMetadata_ondeleteaction_enum`
and set default values.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726757368825-addIsUniqueToIndexMetadata.ts`](diffhunk://#diff-8f1e14bd7f6835ec2c3bb39bcc51e3c318a3008d576a981e682f4c985e746fbfR1-R19):
Added a migration to include the `isUnique` field in `indexMetadata`.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726762935841-addCompostiveColumnToIndexFieldMetadata.ts`](diffhunk://#diff-7c96b7276c7722d41ff31de23b2de4d6e09adfdc74815356ba63bc96a2669440R1-R19):
Added a migration to include the `compositeColumn` field in
`indexFieldMetadata`.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726766871572-addWhereToIndexMetadata.ts`](diffhunk://#diff-26651295a975eb50e672dce0e4e274e861f66feb1b68105eee5a04df32796190R1-R14):
Added a migration to include the `indexWhereClause` field in
`indexMetadata`.
### GraphQL Exception Handling:
*
[`packages/twenty-server/src/engine/api/graphql/workspace-query-runner/utils/workspace-query-runner-graphql-api-exception-handler.util.ts`](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R1-R4):
Enhanced exception handling for `QueryFailedError` to provide more
specific error messages for unique constraint violations.
[[1]](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R1-R4)
[[2]](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R23-R59)
*
[`packages/twenty-server/src/engine/api/graphql/workspace-resolver-builder/factories/create-many-resolver.factory.ts`](diffhunk://#diff-233d58ab2333586dd45e46e33d4f07e04a4b8adde4a11a48e25d86985e5a7943L58-R58):
Updated the `workspaceQueryRunnerGraphqlApiExceptionHandler` call to
include context.
*
[`packages/twenty-server/src/engine/api/graphql/workspace-resolver-builder/factories/create-one-resolver.factory.ts`](diffhunk://#diff-68b803f0762c407f5d2d1f5f8d389655a60654a2dd2394a81318655dcd44dc43L58-R58):
Updated the `workspaceQueryRunnerGraphqlApiExceptionHandler` call to
include context.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Context
Name shouldn't be added to all tables, especially standard objects
because they already have their own labelIdentifierFieldMetadata
specified in the workspace-entity schema. This PR removes this column
from the "base" list of columns to add when creating a new object/table
and moves it to the object-metadata service that is, as of today, only
used for custom objects. Also had to modify the migration-runner to
handle column creation in a table creation migration (this was available
in the migration definition already but was not doing anything)
This also fixes an issue in standard objects that already have a "name"
field defined with a different field type, this is even more important
when the said field is a composite field. For example people already has
a FULL_NAME name field which clashes with the default TEXT name field
meaning it was only creating 1 field metadata for 'name' but 3 columns
were created: `name, nameFirstName, nameLastName`. This inconsistency
with metadata (which is our source of truth everywhere) brought some
issues (lately, converting back typeorm response to gql (including
composition) was broken).
Steps to test
1. Run metadata migrations
2. Run sync-metadata on your workspace
3. Enable the following feature flags:
IS_SEARCH_ENABLED
IS_QUERY_RUNNER_TWENTY_ORM_ENABLED
IS_WORKSPACE_MIGRATED_FOR_SEARCH
4. Type Cmd + K and search anything
The code removed in the PR was flagged as unused by the JetBrains
inspector.
I did a QA on the dev environment but other checks are highly
recommended.
There is one commit by scope to make the review easier.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Context
As we grow, the messaging scripts are experiencing performance issues
forcing us to temporarily disable them on the cloud.
While investigating the performance, I have noticed that generating the
entity schema (for twentyORM) in the repository is taking ~500ms locally
on my Mac M2 so likely more on pods. Caching the entitySchema then!
I'm also clarifying naming around schemaVersion and cacheVersions ==>
both are renamed workspaceMetadataVersion and migrated to the workspace
table (the workspaceCacheVersion table is dropped).
- Remove `messageThreadId` from `messageChannelMessageAssociation`
- Update thread merging
- Update all queries which were dependent on this field
- Update some raw queries by using `twentyORM` instead
---------
Co-authored-by: Weiko <corentin@twenty.com>
This PR introduces the following changes:
- Add the ability to filter webhooks by objectSingularName and Actions
- Refactor SettingsWebhookDetails edition to not use react-hook-form
(which will be deprecated on the whole project)
- Updating the tests with a complex set of mock (we just need to fix ~30
of them now :D)
<img width="1053" alt="image"
src="https://github.com/user-attachments/assets/4e56d972-f129-4789-8d1c-4b5797a8ffd7">
In this PR, I'm:
- adding createdBy field (type ACTOR) on custom objects when created
- moving `name` and `position` default column to the set of columns
automatically creation on object creation
- fixing a bug on mutations (update / create), if the targetted object
has a 'data' custom field, it was conflicting with the payload ==> I
feel we need to refactor this part of the code but we can keep this for
a bit later as we plan to move out of pg_graphql
<img width="1198" alt="image"
src="https://github.com/user-attachments/assets/891c4a97-bab1-415c-8551-dabd5996a794">
When migrating the option values of a select type, if the field is non
nullable (for now, only available for opportunity's "stage" standard
field), we fallback to the (potentially updated) default value instead
of nullifying the value to avoid getting a database error.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
As per title!
Also, I'm removing an incorrect logic in the enum migration runner that
takes care of the case where we have no defaultValue but non nullable
which is not a valid business case.
## Context
Our Flexible Schema engine dynamically generates entities/tables/APIs
for us but was not flexible enough to build indexes in the DB. With more
and more features involving heavy queries such as Messaging, we are now
adding a new WorkspaceIndex() decorator for our standard objects (will
come later for custom objects). This decorator will give enough
information to the workspace sync metadata manager to generate the
proper migrations that will create or drop indexes on demand.
To be aligned with the rest of the engine, we are adding 2 new tables:
IndexMetadata and IndexFieldMetadata, that will store the info of our
indexes.
## Implementation
```typescript
@WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.person,
namePlural: 'people',
labelSingular: 'Person',
labelPlural: 'People',
description: 'A person',
icon: 'IconUser',
})
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
@WorkspaceField({
standardId: PERSON_STANDARD_FIELD_IDS.email,
type: FieldMetadataType.EMAIL,
label: 'Email',
description: 'Contact’s Email',
icon: 'IconMail',
})
@WorkspaceIndex()
email: string;
```
By simply adding the WorkspaceIndex decorator, sync-metadata command
will create a new index for that column.
We can also add composite indexes, note that the order is important for
PSQL.
```typescript
@WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.person,
namePlural: 'people',
labelSingular: 'Person',
labelPlural: 'People',
description: 'A person',
icon: 'IconUser',
})
@WorkspaceIndex(['phone', 'email'])
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```
Currently composite fields and relation fields are not handled by
@WorkspaceIndex() and you will need to use this notation instead
```typescript
@WorkspaceIndex(['companyId', 'nameFirstName'])
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```
<img width="700" alt="Screenshot 2024-06-21 at 15 15 45"
src="https://github.com/twentyhq/twenty/assets/1834158/ac6da1d9-d315-40a4-9ba6-6ab9ae4709d4">
Next step: We might need to implement more complex index expressions,
this is why we have an expression column in IndexMetadata.
What I had in mind for the decorator, still open to discussion
```typescript
@WorkspaceIndex(['nameFirstName', 'nameLastName'], { expression: "$1 || ' ' || $2"})
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Update of select fields options was failing if we deleted an option that
was used for at least one row: former code would not update the value to
null but leave it to the no-longer-allowed value.
Closes#5069 back-end part
And:
- do not display schemaPendingUpdates status on remote server lists as
this call will become too costly if there are dozens of servers
- (refacto) create foreignTableService
After this is merged we will be able to delete remoteTable's
availableTables column
Fixes#5276.
Updates were not triggering a cache version incrementation because they
do not trigger migrations while that is where the caching version logic
was.
We have decided to move the cache incrementation logic to the services.
This PR fixes several issues:
- enum naming should be: {tableName}_{fieldName}_enum and respecting the
case
- defaultValue format handled in the FE should respect the one in the BE
In my opinion we should refactor the defaultValue:
- we should respect backend format: "'myDefault'" for constant default
and "0" for float, "now" for expressions, "true" for booleans. we can
rename it to defaultValueExpression if it is more clear but we should
not maintain a parallel system
- we should deprecate option: isDefaultValue which is confusing
- we should re-work backend to have a more unified approach between
fields and avoid having if everywhere about select, multiselect, and
currency cases. one unified "computeDefaultValue" function should do the
job
What is still broken:
- currency default Value on creation. I think we should do the refactor
first
- select default value edition.
These cases do not break the schema but are ignored currently
Various fixes
- Remote objects are read-only for now, we already hide and block most
of the write actions but the button that allows you to add a new record
in an empty collection was still visible.
- CreatedAt is not mandatory on remote objects (at least for now) so it
was breaking the show page, it now checks if createdAt exists and is not
null before trying to display the human readable format `Added x days
ago`
- The filters are overwritten in query-runner-args.factory.ts to handle
NUMBER field type, this was only working with filters like
```
{
"id": {
"in": [
1
]
}
```
but not with more depth such as
```
"and": [
{},
{
"id": {
"in": [
1
]
}
}
]
```
- Fixes CREATE FOREIGN TABLE raw query which was missing ",".
Adding stripe integration by making the server logic independent of the
input fields:
- query factories (remote server, foreign data wrapper, foreign table)
to loop on fields and values without hardcoding the names of the fields
- adding stripe input and type
- add the logic to handle static schema. Simply creating a big object to
store into the server
Additional work:
- rename username field to user. This is the input intended for postgres
user mapping and we now need a matching by name
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
We should not depend on the foreign data wrapper type to manage distant
table. The remote server should be enough to handle the table creation.
Here is the new flow to fetch available tables:
- check if the remote server have available tables already stored
- if not, import full schema in a temporary schema
- copy the tables into the available tables field
- delete the schema
Left todo:
- update remote server input for postgres so we receive the schema
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
We've introduced in PR #4373 standard ids to be able to rename standard
fields and objects.
Fields part was working properly, but objects part was not yet
implemented.
This PR is adding the missing parts to make it work.
Foreign tables should be created using migrations, as we do for standard
tables.
Since those are not really generated from the object metadata but from
the remote table, those migrations won't live in the object metadata
service.
This PR:
- creates new types of migration : create_foreign_table and
drop_foreign_table
- triggers those migrations rather than raw queries directly
- moves the logic to fetch current foreign tables into the remote table
service since this is not directly linked to postgres data wrapper
- adds logic to unsync all tables before deleting
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
We've seen a few cascading errors (e.g. comment.activityId would be non
nullable but cascade behavior is set to "set null"). I think it's safer
if we have to explicitly chose the deletion behavior it every time.
Especially since Postgres default to "No action" while we defaulted to
"Set Null", which is confusing.
In the future we will most likely introduce a second param
`onSoftDelete` in the decorator
Foreign table id cannot be a foreign key of a base table. But the
current code use foreign keys to link object metadata with activities,
events... So we will:
- create a column without creating a foreign key
- add a comment on the table schema so pg_graphql sees it as a foreign
key
This PR:
- refactor a bit object metadata service so the mutation creation is
separated into an util
- adds the mutation creation for remote object relations
- add a new type of mutation to create a comment
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
* fix: sever not throwing when enum contains two identical values
* fix: enum column name cannot be change
* fix: put field create/update inside transactions
* fix: check for options duplicate values front-end
* fix: missing commit transaction
