Fixes https://github.com/twentyhq/twenty/issues/8810
Fixes https://github.com/twentyhq/twenty/issues/5268
Fixes https://github.com/twentyhq/twenty/issues/8971
- Fixing Task/Note creation not sending position during creation
- Adding a command to backfill position being null, using existing
backfill command.
- Removed unused backfill job.
- Updated workspace entities to set position non-nullable and set a
default value to make it non-required on the API
- Updated position factory to set a default position for all objects
having a POSITION field instead of only company/people
- Moved the try/catch in each resolver factory calling
GraphqlQueryRunnerException handler, makes more sense to call it in the
actual graphql-query-runner and removing some duplicate codes
- Adding validations for input in QueryRunnerArgs factories
- Allow sync-metadata to override and sync defaultValues for certain
field types (that can't be updated by users)
- Removing health-check from sync-metadata command during force mode to
improve performances
## Context
The recent addition of object renaming introduced issues with enum
names. Enum names should follow the pattern
`${schemaName}.${tableName}_${columnName}_enum`. To address this, and to
allow users to customize the API name (which is included in the enum
name, columnName), this PR implements behavior similar to object
renaming by introducing a `isLabelSyncedWithName` boolean.
<img width="624" alt="Screenshot 2024-12-02 at 11 58 49"
src="https://github.com/user-attachments/assets/690fb71c-83f0-4922-80c0-946c92dacc30">
<img width="596" alt="Screenshot 2024-12-02 at 11 58 39"
src="https://github.com/user-attachments/assets/af9a0037-7cf5-40c3-9ed5-d51b340c8087">
## Context
Enum are named after this pattern
`${schema}_${tableName}_${columnName}_enum` however now that we allowed
table name update, we need to make sure their enums are renamed as well.
## Issue
Some users were facing issues while updating SELECT or MULTISELECT
options:
```
Error executing migration QueryFailedError: column "undefined" does not exist
at PostgresQueryRunner.query (/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async WorkspaceMigrationEnumService.migrateEnumValues (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/services/workspace-migration-enum.service.js:101:13)
at async WorkspaceMigrationEnumService.alterEnum (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/services/workspace-migration-enum.service.js:54:9)
at async WorkspaceMigrationRunnerService.alterColumn (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:254:13)
at async WorkspaceMigrationRunnerService.handleColumnChanges (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:202:21)
at async WorkspaceMigrationRunnerService.handleTableChanges (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:94:25)
at async WorkspaceMigrationRunnerService.executeMigrationFromPendingMigrations (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:60:17)
at async WorkspaceExecutePendingMigrationsCommand.run (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/commands/workspace-execute-pending-migrations.command.js:24:9)
at async Command.<anonymous> (/app/node_modules/nest-commander/src/command-runner.service.js:162:24) {
query: '\n' +
' UPDATE "workspace_7i7bwawp7keh3gel1g69jropc"."_funnelInProductsRu"\n' +
' SET "reasonForStageTransition" = undefined\n' +
" WHERE id='2af8db61-5f75-46de-8b1a-97e312937e06'\n" +
' ',
parameters: undefined,
driverError: error: column "undefined" does not exist
```
## Root cause
Internally, while migrating enum values, we are doing:
```
const values = await queryRunner.query(
`SELECT id, "${oldColumnName}" FROM "${schemaName}"."${tableName}"`,
);
let val = value[oldColumnName];
```
oldColumnName being: `${columnDefinition.columnName}_old_${v4()}`;
However, TypeORM only supports 63 bits identifiers:
https://github.com/typeorm/typeorm/issues/3118, but silently truncate
the identifer so if oldColumnName gets larger than 63 characters,
`value[oldColumnName]` is undefined.
## Fix
We only need a temporary and unique temporary name, no need to take into
account the previous columnName here
## Context
- Fixing folder structure where 0.33 was inside 0.32.
- Updating the command so it runs enforce uniqueness on all tables
- Updating workspaces entities so the sync metadata adds the index
Search vector fields based on `RICH_TEXT` fields were generated using a
treatment on `RICH_TEXT` fields's `body`
column, to only extract and index the core text.
([PR](https://github.com/twentyhq/twenty/pull/7953))
Actually our RICH_TEXT fields are of datatype `text` in our database,
allowing users to insert non-json values, which breaks the search vector
generation (as it expects json values).
Our vision is unclear for now: for instance we may want to turn
RICH_TEXT into a composite field where the plain text would be stored in
a different column.
So for now, we will (1) treat rich_text data as text, and (2) update the
search vector expressions for the existing workspaces.
- Removing unique constraint creation in 0.32 as we have a dependency on
sync-metadata and sync-metadata has a dependency on it if we keep it.
- sync-metadata does not support well the deletion of activity +
activityTarget, silently swallow the exception for now
## Context
ObjectMetadataService became quite large and handles too many
responsibilities. I'm trying to refactor a bit this part in preparation
of a larger work that will combine object-metadata services and
sync-metadata logic
- Created a STANDARD_OBJECT_ICONS that can be reused in relation
creation to refer to a standard object icon.
- Created a STANDARD_OBJECT_FIELD_IDS that can be used with an object
name to get its standard field ids.
- Moved migration, record and relation creations to dedicated services,
refactored to improve performances and readability
- Refactored some validation logic
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Fix#4244 and #4356
This pull request introduces the new "view groups" capability, enabling
the reordering, hiding, and showing of columns in Kanban mode. The core
enhancement includes the addition of a new entity named `ViewGroup`,
which manages column behaviors and interactions.
#### Key Changes:
1. **ViewGroup Entity**:
The newly added `ViewGroup` entity is responsible for handling the
organization and state of columns.
This includes:
- The ability to reorder columns.
- The option to hide or show specific columns based on user preferences.
#### Conclusion:
This PR adds a significant new feature that enhances the flexibility of
Kanban views through the `ViewGroup` entity.
We'll later add the view group logic to table view too.
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
`isUnique` was passed to TypeORM's column creation API resulting in
double index creation because it's already done via the decorator and
then in `WorkspaceMigrationIndexFactory`
It would be interesting to move it at this field level in a later step,
which is why I also fixed `CompositeColumnActionFactory` to pass
isUnique on the correct columns, even though it's being ignored later on
By default, when custom fields are created, a searchVector field is
created based on the "name" field, which is also the label identifier by
default.
When this label identifier is updated, we want to update the
searchVector field to use this field as searchable field instead, if it
is of "searchable type" (today it is only possible to select a text or
number field as label identifier, while number fields are not
searchable).
## Context
This PR removes workspace-query-runner/builder in preparation for fully
deprecating pg_graphql
next steps: Remove from the setup and make a command to remove comments
on schema/tables related to pg_graphql
Add support for indexes on composite fields and unicity constraint on
indexes
This pull request includes several changes across multiple files to
improve error handling, enforce unique constraints, and update database
migrations. The most important changes include updating error messages
for snack bars, adding a new command to enforce unique constraints, and
updating database migrations to include new fields and constraints.
### Error Handling Improvements:
*
[`packages/twenty-front/src/modules/error-handler/components/PromiseRejectionEffect.tsx`](diffhunk://#diff-e7dc05ced8e4730430f5c7fcd0c75b3aa723da438c26e0bef8130b614427dd9aL23-R23):
Updated error messages in `enqueueSnackBar` to use `error.message`
directly.
*
[`packages/twenty-front/src/modules/object-metadata/hooks/useFindManyObjectMetadataItems.ts`](diffhunk://#diff-74c126d6bc7a5ed6b63be994d298df6669058034bfbc367b11045f9f31a3abe6L44-R46):
Simplified error messages in `enqueueSnackBar`.
*
[`packages/twenty-front/src/modules/object-record/hooks/useFindDuplicateRecords.ts`](diffhunk://#diff-af23a1d99639a66c251f87473e63e2b7bceaa4ee4f70fedfa0fcffe5c7d79181L56-R58):
Simplified error messages in `enqueueSnackBar`.
*
[`packages/twenty-front/src/modules/object-record/hooks/useHandleFindManyRecordsError.ts`](diffhunk://#diff-da04296cbe280202a1eaf6b1244a30490d4f400411bee139651172c59719088eL22-R24):
Simplified error messages in `enqueueSnackBar`.
### New Command for Unique Constraints:
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-enforce-unique-constraints.command.ts`](diffhunk://#diff-8337096c8c80dd2619a5ba691ae5145101f8ae0368a75192a050047e8c6ab7cbR1-R159):
Added a new command to enforce unique constraints on company domain
names and person emails.
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-upgrade-version.command.ts`](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR13-R14):
Integrated the new `EnforceUniqueConstraintsCommand` into the upgrade
process.
[[1]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR13-R14)
[[2]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR31)
[[3]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR64-R68)
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-upgrade-version.module.ts`](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R7):
Registered the new `EnforceUniqueConstraintsCommand` in the module.
[[1]](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R7)
[[2]](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R24)
### Database Migrations:
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726757368824-migrationDebt.ts`](diffhunk://#diff-c450aeae7bc0ef4416a0ade2dc613ca3f688629f35d2a32f90a09c3f494febdcR1-R53):
Added a migration to update the `relationMetadata_ondeleteaction_enum`
and set default values.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726757368825-addIsUniqueToIndexMetadata.ts`](diffhunk://#diff-8f1e14bd7f6835ec2c3bb39bcc51e3c318a3008d576a981e682f4c985e746fbfR1-R19):
Added a migration to include the `isUnique` field in `indexMetadata`.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726762935841-addCompostiveColumnToIndexFieldMetadata.ts`](diffhunk://#diff-7c96b7276c7722d41ff31de23b2de4d6e09adfdc74815356ba63bc96a2669440R1-R19):
Added a migration to include the `compositeColumn` field in
`indexFieldMetadata`.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726766871572-addWhereToIndexMetadata.ts`](diffhunk://#diff-26651295a975eb50e672dce0e4e274e861f66feb1b68105eee5a04df32796190R1-R14):
Added a migration to include the `indexWhereClause` field in
`indexMetadata`.
### GraphQL Exception Handling:
*
[`packages/twenty-server/src/engine/api/graphql/workspace-query-runner/utils/workspace-query-runner-graphql-api-exception-handler.util.ts`](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R1-R4):
Enhanced exception handling for `QueryFailedError` to provide more
specific error messages for unique constraint violations.
[[1]](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R1-R4)
[[2]](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R23-R59)
*
[`packages/twenty-server/src/engine/api/graphql/workspace-resolver-builder/factories/create-many-resolver.factory.ts`](diffhunk://#diff-233d58ab2333586dd45e46e33d4f07e04a4b8adde4a11a48e25d86985e5a7943L58-R58):
Updated the `workspaceQueryRunnerGraphqlApiExceptionHandler` call to
include context.
*
[`packages/twenty-server/src/engine/api/graphql/workspace-resolver-builder/factories/create-one-resolver.factory.ts`](diffhunk://#diff-68b803f0762c407f5d2d1f5f8d389655a60654a2dd2394a81318655dcd44dc43L58-R58):
Updated the `workspaceQueryRunnerGraphqlApiExceptionHandler` call to
include context.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Context
Because we can't create GIN indexes on primitive types and in this case
both tsvector and deletedAt (date), we had to create an index on the
tsvector only and had to make sure it wouldn't return results for
deleted records.
To handle this, we added a CASE in the generated expression to generate
an empty value if the row has been soft-deleted. This is a bit too
complex and won't allow search on softDeleted records in the future.
What we want to do is to make sure deleted records are not returned by
default by adding a WHERE clause in the search and still keep good
performances by also adding a BTREE index on deletedAt column.
When this was implemented, soft-deleted was not handled properly but now
typeorm query builder exclude soft deleted records by default which
means the WHERE clause is not necessary. As for the BTREE index on
deletedAt, this should be part on a broader effort to add it to all
tables and can be added in a later PR.
This PR simply updates the complex tsVectorExpression to only return the
expression regardless of the deletedAt column. Search won't return
soft-deleted records as explained above and perfs will be improved in an
upcoming version.
## Test
tested locally with reset db + created a new workspace. Since the
feature is not launched yet no more effort should be made.
<img width="1289" alt="Screenshot 2024-10-10 at 15 16 16"
src="https://github.com/user-attachments/assets/d9a6f06b-d738-4f1d-8053-2fa93bd8b4f8">
## Note
Some issues with optimistic rendering, we need to refresh once a record
has been deleted otherwise it will still be returned by the search
- update `send-email.workflow-action.ts` so it send email via the google
sdk
- remove useless `workflow-action.email.ts`
- add `send` authorization to google api scopes
- update the front workflow email step form to provide a
`connectedAccountId` from the available connected accounts
- update the permissions of connected accounts: ask users to reconnect
when selecting missing send permission
Fix syncedAt no longer been set on message sync.
Fix calendar data model:
- Add `syncedAt` to `CalendarChannelWorkspaceEntity`
- Move `recurringEventExternalId` from `CalendarEventWorkspaceEntity` to
`CalendarChannelEventAssociationWorkspaceEntity` since the id is
relative to one channel
Fix save queries on calendar sync after regression.
Enabling feature flags IsSearchEnabled and IsWorkspaceMigratedForSearch
at workspace creation to ensure workspaces have the searchVector fields
and indexes created.
For the feature to be enabled in the front-end we will also need
IsQueryRunnerTwentyORMEnabled to be enabled but that is an independent
topic.
When object is not part of the workspace favorite list, we want to show
it in the "opened section" while its record page is accessed.
This PR:
- adds a new component `NavigationDrawerOpenedSection`
- makes workflow versions and runs not system object + creates a
prefilled view index for these
- do not create workspace favorites for these so these do not appear in
the workspace section
<img width="1129" alt="Capture d’écran 2024-09-26 à 11 45 25"
src="https://github.com/user-attachments/assets/c84d773c-0bef-4dce-b66a-55d7d00b0fb6">
## Context
Name shouldn't be added to all tables, especially standard objects
because they already have their own labelIdentifierFieldMetadata
specified in the workspace-entity schema. This PR removes this column
from the "base" list of columns to add when creating a new object/table
and moves it to the object-metadata service that is, as of today, only
used for custom objects. Also had to modify the migration-runner to
handle column creation in a table creation migration (this was available
in the migration definition already but was not doing anything)
This also fixes an issue in standard objects that already have a "name"
field defined with a different field type, this is even more important
when the said field is a composite field. For example people already has
a FULL_NAME name field which clashes with the default TEXT name field
meaning it was only creating 1 field metadata for 'name' but 3 columns
were created: `name, nameFirstName, nameLastName`. This inconsistency
with metadata (which is our source of truth everywhere) brought some
issues (lately, converting back typeorm response to gql (including
composition) was broken).
Steps to test
1. Run metadata migrations
2. Run sync-metadata on your workspace
3. Enable the following feature flags:
IS_SEARCH_ENABLED
IS_QUERY_RUNNER_TWENTY_ORM_ENABLED
IS_WORKSPACE_MIGRATED_FOR_SEARCH
4. Type Cmd + K and search anything
This PR was created by [GitStart](https://gitstart.com/) to address the
requirements from this ticket:
[TWNTY-6260](https://clients.gitstart.com/twenty/5449/tickets/TWNTY-6260).
This ticket was imported from:
[TWNTY-6260](https://github.com/twentyhq/twenty/issues/6260)
---
### Description
This is the second PR on TWNTY-6260 which handles data migration of
Phone field to Phones field.\
\
How to Test?\
Follow the below steps:
- On the main branch,
- go to
`packages/twenty-server/src/database/typeorm-seeds/workspace/people.ts`
and change any person's phone number to a string with characters for
example: "test invalid phone", and then reset the DB.
- reset database using `npx nx database:reset twenty-server`
- This is to make sure that invalid numbers will be handled properly. We
should use the invalid value itself to avoid removing data and see how
the behavior is on the front end. should be the same as in the main, the
display shows the invalid value, but the input is empty when you click,
and then you can update.
- Checkout to `TWNTY-6260-phone-migration` branch
- Rebuild typescript using `npx nx build twenty-server`
- Run command `yarn command:prod upgrade-0.32` to do migration
- Run both backend and frontend to see the migrated field
### Demo
- **Loom Video:**\
<https://www.loom.com/share/4b9bcb423cee447d8ad09852a83b27da?sid=ed74ecaa-0339-4575-acdc-a863e95e94fd>
### Refs
#6260
---------
Co-authored-by: gitstart-twenty <gitstart-twenty@users.noreply.github.com>
Co-authored-by: Marie Stoppa <marie.stoppa@essec.edu>
Co-authored-by: Weiko <corentin@twenty.com>
- Improve snackbar to enable displaying multi-line message (so far we
only displayed the first few words which was very frustrating)
- Followup on previous issue to enable tim@apple.dev on the demo
workspace (prefilled automatically)
- Fix sentry tracing which had been broken when migrating from v7 to v8
In this PR:
- removing ugprade-0.24 commands as we are releasing 0.30
- introducing cache:flush command
- refactoring upgrade command and sync-metadata command to use the
ActiveWorkspacesCommand so they consistently run on all workspaces or
selected workspaces
Fixes:
- clear localStorage on sign out
- fix missing workspaceMember in verify resolver
- do not throw on datasource already destroyed exception which can
happen with race condition when several resolvers are resolving in
parallel
This https://github.com/twentyhq/twenty/pull/7006 introduced a
regression.
The goal was to set "isSoftDeletable" to all standard objects but it was
done at the wrong level, meaning it was setting the boolean correctly
but not creating the corresponding fieldMetadata.
I took the occasion to update the new graphql query runner to use that
boolean and automatically add a filter on soft delete in case it's true.
Also adding **IsQueryRunnerTwentyORMEnabled** by default in the seeds
This is the second PR on TWNTY-6261 which handlesdata migration of Email
field to Emails field.\
\
How to Test?\
Firstly make sure that you have completed the testing steps on first PR
then follow the below steps:
- Checkout to TWNTY-6261-emails-migrations branch
- Rebuild typescript using "npx nx build twenty-server"
- Run command "yarn command:prod upgrade-0.25" to do migration\
\
Loom Video:\
<https://www.loom.com/share/f82b8d29f8f64f92abe3c59c01147b45?sid=9f8ccc05-aa38-4c49-b139-fd0823066273>
**Testing Messaging Sync functionality:**
Please watch the below video to see that the synchronization of contacts
is working fine after migrating Email field to Emails field:\
<https://www.loom.com/share/400949464b244272b78c25e338cc6ab2?sid=103f6625-5933-4b99-9825-0fed33782f36>
**Question to the client**
should we rename email to emails here? in the DomainName PR, the name
did not change.
```typescript
@WorkspaceField({
standardId: PERSON_STANDARD_FIELD_IDS.email,
type: FieldMetadataType.EMAILS,
label: 'Email',
description: 'Contact’s Email',
icon: 'IconMail',
})
email: EmailsMetadata;
```
**Test Messaging Sync**
This pr will update messaging sync files so the changes shouldn't break
existing functionality of importing people and companies in the app.\
To test messaging sync you should follow the below steps:\
1. you need to connect a google account to see the importing
functionality. For this purpose you
have to create a project inside Google Cloud. But to make things easier
you can use the below credentials of an already created project. Put
them in .env of twenty-server package:
```properties
MESSAGING_PROVIDER_GMAIL_ENABLED=true
CALENDAR_PROVIDER_GOOGLE_ENABLED=true
AUTH_GOOGLE_ENABLED=true
AUTH_GOOGLE_CLIENT_ID=951231465939-h61tg6nkpkv1821qi899fjbj9looquto.apps.googleusercontent.com
AUTH_GOOGLE_CLIENT_SECRET=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
AUTH_GOOGLE_CALLBACK_URL=http://localhost:3000/auth/google/redirect
AUTH_GOOGLE_APIS_CALLBACK_URL=http://localhost:3000/auth/google-apis/get-access-token
MESSAGE_QUEUE_TYPE=bull-mq
```
Alternative env
```properties
MESSAGING_PROVIDER_GMAIL_ENABLED=true
CALENDAR_PROVIDER_GOOGLE_ENABLED=true
AUTH_GOOGLE_ENABLED=true
AUTH_GOOGLE_CLIENT_ID=622006708006-dc4n3vrtf3cs2h6k7hgbborudme7ku9l.apps.googleusercontent.com
AUTH_GOOGLE_CLIENT_SECRET=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
AUTH_GOOGLE_CALLBACK_URL=http://localhost:3000/auth/google/redirect
AUTH_GOOGLE_APIS_CALLBACK_URL=http://localhost:3000/auth/google-apis/get-access-token
MESSAGE_QUEUE_TYPE=bull-mq
```
1. Launch your worker with `npx nx run twenty-server:worker`
2. npx nx run twenty-server:command cron:messaging:messages-import
3. npx nx run twenty-server:command cron:messaging:message-list-fetch
4. npx nx run twenty-server:command
cron📆calendar-event-list-fetch
5. Run the app and navigate to Settings/Accounts then connect your
Google account
---------
Co-authored-by: gitstart-twenty <gitstart-twenty@users.noreply.github.com>
Co-authored-by: Marie Stoppa <marie.stoppa@essec.edu>
Co-authored-by: Weiko <corentin@twenty.com>
This PR introduces the following changes:
- add the metadataVersion to all our metadata cache keys to ease
troubleshooting:
<img width="1146" alt="image"
src="https://github.com/user-attachments/assets/8427805b-e07f-465e-9e69-1403652c8b12">
- introduce a cache recompute lock to avoid overloading the database to
recompute the cache many time
- make member nullable on favorites
- add potential relation with view entity
- add a new type of favorite list in front : workspace favorite
- build a new component for retrieving workspace favorite to display +
refacto the existing one
Bonus:
- removing activities seed since this is deprecated
