In the expression of our searchVector fields, we use the "simple"
configuration (over the default "english" one), to avoid picking a
language that's irrelevant to the user.
I initially forgot to add the same configuration to the query that is
being sent using ts_query.
Adding it will also allow the search to work for a single character,
while so far a single letter was most of the time considered a "stop
word" (a word with no semantic value, like "a").
- fix webhook.operation format change from august 2024 not spread in
twenty-zapier
- added a comment so it does not happen again
- add a fix for the new webhook.operations column that would produce
another issue
The recent addition of a "orWhere" condition to[ improve the search algo
quality](https://github.com/twentyhq/twenty/pull/7955) accidentally
broke the filter, being considered an independent "or" wondition while
we still want the filter to apply.
Solves https://github.com/twentyhq/private-issues/issues/118
**TLDR**
Fix webhook response not sending data to tinybird when the url is not a
link.
**Changes in Tinybird:**
- Add column Success to webhook payload (boolean)
- Changed the parameter WebhookIdRequest to WebhookId in the
getWebhooksResponse api point.
- Those changes can be seen in the tinybird workspace
twenty_analytics_playground
**In order to test**
1. Set ANALYTICS_ENABLED to true
2. Set TINYBIRD_INGEST_TOKEN to your token from the workspace
twenty_analytics_playground
3. Set TINYBIRD_GENERATE_JWT_TOKEN to the admin kwt token from the
workspace twenty_analytics_playground
4. Set TINYBIRD_WORKSPACE_UUID to the UUID of
twenty_analytics_playground
5. Create a Webhook in twenty and set wich events it needs to track
6. Run twenty-worker in order to make the webhooks work.
7. Do your tasks in order to populate the data
8. Look at your webhooks in settings>api and webhooks> your webhook and
the statistics should be displayed
We were previously checking for matching with each search term
independently. Ex searching for "felix malfait" we were searching for
correspondances with "felix" and "malfait".
As a result record A with name "Marie-Claude Mala" and email
"ma.lala@email.com" had a biggest search score than record B "Felix
Malfait" with email felix@email.com for search "felix ma":
for record A we had 0 match with felix and 3 matches with "ma" ("marie",
"mala", "ma")
for record B we had 1 match with felix and 1 match with "ma" (with
"malfait").
So we want to give more weight to a row that would combine matches with
both terms, considering "felix malfait" altogether.
First step of #https://github.com/twentyhq/twenty/issues/3298.
Here we update the search endpoint to allow for a filter argument, which
we currently use in the relation pickers to restrict or exclude ids from
search.
In a future PR we will try to simplify the search logic in the FE
By default, when custom fields are created, a searchVector field is
created based on the "name" field, which is also the label identifier by
default.
When this label identifier is updated, we want to update the
searchVector field to use this field as searchable field instead, if it
is of "searchable type" (today it is only possible to select a text or
number field as label identifier, while number fields are not
searchable).
## Context
This PR removes workspace-query-runner/builder in preparation for fully
deprecating pg_graphql
next steps: Remove from the setup and make a command to remove comments
on schema/tables related to pg_graphql
Add support for indexes on composite fields and unicity constraint on
indexes
This pull request includes several changes across multiple files to
improve error handling, enforce unique constraints, and update database
migrations. The most important changes include updating error messages
for snack bars, adding a new command to enforce unique constraints, and
updating database migrations to include new fields and constraints.
### Error Handling Improvements:
*
[`packages/twenty-front/src/modules/error-handler/components/PromiseRejectionEffect.tsx`](diffhunk://#diff-e7dc05ced8e4730430f5c7fcd0c75b3aa723da438c26e0bef8130b614427dd9aL23-R23):
Updated error messages in `enqueueSnackBar` to use `error.message`
directly.
*
[`packages/twenty-front/src/modules/object-metadata/hooks/useFindManyObjectMetadataItems.ts`](diffhunk://#diff-74c126d6bc7a5ed6b63be994d298df6669058034bfbc367b11045f9f31a3abe6L44-R46):
Simplified error messages in `enqueueSnackBar`.
*
[`packages/twenty-front/src/modules/object-record/hooks/useFindDuplicateRecords.ts`](diffhunk://#diff-af23a1d99639a66c251f87473e63e2b7bceaa4ee4f70fedfa0fcffe5c7d79181L56-R58):
Simplified error messages in `enqueueSnackBar`.
*
[`packages/twenty-front/src/modules/object-record/hooks/useHandleFindManyRecordsError.ts`](diffhunk://#diff-da04296cbe280202a1eaf6b1244a30490d4f400411bee139651172c59719088eL22-R24):
Simplified error messages in `enqueueSnackBar`.
### New Command for Unique Constraints:
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-enforce-unique-constraints.command.ts`](diffhunk://#diff-8337096c8c80dd2619a5ba691ae5145101f8ae0368a75192a050047e8c6ab7cbR1-R159):
Added a new command to enforce unique constraints on company domain
names and person emails.
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-upgrade-version.command.ts`](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR13-R14):
Integrated the new `EnforceUniqueConstraintsCommand` into the upgrade
process.
[[1]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR13-R14)
[[2]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR31)
[[3]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR64-R68)
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-upgrade-version.module.ts`](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R7):
Registered the new `EnforceUniqueConstraintsCommand` in the module.
[[1]](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R7)
[[2]](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R24)
### Database Migrations:
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726757368824-migrationDebt.ts`](diffhunk://#diff-c450aeae7bc0ef4416a0ade2dc613ca3f688629f35d2a32f90a09c3f494febdcR1-R53):
Added a migration to update the `relationMetadata_ondeleteaction_enum`
and set default values.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726757368825-addIsUniqueToIndexMetadata.ts`](diffhunk://#diff-8f1e14bd7f6835ec2c3bb39bcc51e3c318a3008d576a981e682f4c985e746fbfR1-R19):
Added a migration to include the `isUnique` field in `indexMetadata`.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726762935841-addCompostiveColumnToIndexFieldMetadata.ts`](diffhunk://#diff-7c96b7276c7722d41ff31de23b2de4d6e09adfdc74815356ba63bc96a2669440R1-R19):
Added a migration to include the `compositeColumn` field in
`indexFieldMetadata`.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726766871572-addWhereToIndexMetadata.ts`](diffhunk://#diff-26651295a975eb50e672dce0e4e274e861f66feb1b68105eee5a04df32796190R1-R14):
Added a migration to include the `indexWhereClause` field in
`indexMetadata`.
### GraphQL Exception Handling:
*
[`packages/twenty-server/src/engine/api/graphql/workspace-query-runner/utils/workspace-query-runner-graphql-api-exception-handler.util.ts`](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R1-R4):
Enhanced exception handling for `QueryFailedError` to provide more
specific error messages for unique constraint violations.
[[1]](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R1-R4)
[[2]](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R23-R59)
*
[`packages/twenty-server/src/engine/api/graphql/workspace-resolver-builder/factories/create-many-resolver.factory.ts`](diffhunk://#diff-233d58ab2333586dd45e46e33d4f07e04a4b8adde4a11a48e25d86985e5a7943L58-R58):
Updated the `workspaceQueryRunnerGraphqlApiExceptionHandler` call to
include context.
*
[`packages/twenty-server/src/engine/api/graphql/workspace-resolver-builder/factories/create-one-resolver.factory.ts`](diffhunk://#diff-68b803f0762c407f5d2d1f5f8d389655a60654a2dd2394a81318655dcd44dc43L58-R58):
Updated the `workspaceQueryRunnerGraphqlApiExceptionHandler` call to
include context.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
In this PR, I'm fixing part of the impact of soft deletion on optimistic
rendering.
## Backend Vision
1) Backend endpoints will not return soft deleted records (having
deletedAt set) by default. To get the softDeleted records, we will pass
a { withSoftDelete: true } additional param in the query.
2) Record relations will NEVER contain softDeleted relations
## Backend current state
Right now, we have the following behavior:
- if the query filters do not mention deletedAt, we don't return
softDeletedRecords
- if the query filters mention deletedAt, we take it into consideration.
Meaning that if we want to have the softDeleted records in any way we
need to do { or: [ deletedAt: NULL, deletedAt: NOT_NULL] }
## Optimistic rendering strategy
1) useDestroyOne/Many is triggering destroyOptimisticEffects (previously
deleteOptimisticEffects)
2) UseDeleteOne/Many and useRestoreOne/Many are actually triggering
updateOptimisticEffects (as they only update deletedAt field) AND we
need updateOptimisticEffects to take into account deletedAt (future
withSoftDelete: true) filter.
While rebuilding the new GraphQLAPI (without pg_graphql), we forgot to
include the FieldGetter logic. This logic will soon be moved at ORM
level but we will need to keep it there for now
The bug has many impacts such as the fileToken not being generated and
preventing users from loading files
In this PR:
- remove deprecated EMAIL, PHONE, LINK field types (except for Zapier
package as there is another work ongoing)
- remove composite currency filter on currencyCode, actor filter on name
and workspaceMember as the UX is not great yet
#7346#7343#7342#7344
Before:
<img width="799" alt="Screenshot 2024-10-08 at 11 59 37"
src="https://github.com/user-attachments/assets/a1cd1714-41ed-4f96-85eb-2861e7a8b2c2">
Now:
In order to test:
1. Set ANALYTICS_ENABLED to true
2. Set TINYBIRD_TOKEN to your token from the workspace
_twenty_analytics_playground_
3. Write your client tinybird token in
SettingsDeveloppersWebhookDetail.tsx in line 93
4. Create a Webhook in twenty and set wich events it needs to track
5. Run twenty-worker in order to make the webhooks work.
6. Do your tasks in order to populate the data
7. Enter to settings> webhook>your webhook and the statistics section
should be displayed.
Fix all the broken CIs :p
This includes an ongoing effort to simplify test maintenance by having 1
unique source of truth about metadata and data mocks (that will later be
generated from a unique source of seeds: dev = demo = test)
Regressions:
- Unit line coverage: 60 > 55
- Storybook Pages branch coverage: 40 > 35
We will need to write tests to increase those coverage
- RelationFieldDisplay perf: 0.2ms to 0.22ms > We might have a
regression here
- Removed perf story about RawJSON > We will need to re-add it
## Context
Name shouldn't be added to all tables, especially standard objects
because they already have their own labelIdentifierFieldMetadata
specified in the workspace-entity schema. This PR removes this column
from the "base" list of columns to add when creating a new object/table
and moves it to the object-metadata service that is, as of today, only
used for custom objects. Also had to modify the migration-runner to
handle column creation in a table creation migration (this was available
in the migration definition already but was not doing anything)
This also fixes an issue in standard objects that already have a "name"
field defined with a different field type, this is even more important
when the said field is a composite field. For example people already has
a FULL_NAME name field which clashes with the default TEXT name field
meaning it was only creating 1 field metadata for 'name' but 3 columns
were created: `name, nameFirstName, nameLastName`. This inconsistency
with metadata (which is our source of truth everywhere) brought some
issues (lately, converting back typeorm response to gql (including
composition) was broken).
Fixes https://github.com/twentyhq/twenty/issues/6859
This PR adds all the remaining resolvers for
- updateOne/updateMany
- createOne/createMany
- deleteOne/deleteMany
- destroyOne
- restoreMany
Also
- refactored the graphql-query-runner to be able to add other resolvers
without too much boilerplate.
- add missing events that were not sent anymore as well as webhooks
- make resolver injectable so they can inject other services as well
- use objectMetadataMap from cache instead of computing it multiple time
- various fixes (mutation not correctly parsing JSON, relationHelper
fetching data with empty ids set, ...)
Next steps:
- Wrapping query builder to handle DB events properly
- Move webhook emitters to db event listener
- Add pagination where it's missing (findDuplicates, nested relations,
etc...)
Steps to test
1. Run metadata migrations
2. Run sync-metadata on your workspace
3. Enable the following feature flags:
IS_SEARCH_ENABLED
IS_QUERY_RUNNER_TWENTY_ORM_ENABLED
IS_WORKSPACE_MIGRATED_FOR_SEARCH
4. Type Cmd + K and search anything
Solves issue #5917.
This PR is now ready for the first review!
Filters do not fully work yet, there's a problem applying multiple
filters like the following:
```
{
and: [
{
[correspondingField.name]: {
gte: start.toISOString(),
} as DateFilter,
},
{
[correspondingField.name]: {
lte: end.toISOString(),
} as DateFilter,
},
],
}
```
I'll do my best to dig into it tonight!
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
#7091
EventTrackers send information of events to the TinyBird instance:
In order to test:
1. Set ANALYTICS_ENABLED= true and TELEMETRY_ENABLED=true in
evironment-variables.ts
2. Set the TINYBIRD_TOKEN in environment variables (go to TiniyBird
Tokens)
3. Log in to twenty's TinyBird and go to datasources/analytics_events in
twenty_analytics workspace
4. Run twenty and navigate it
5. New events will be logged in the datasources, containing their
timestamp, sessionId and payload.
<img width="1189" alt="Screenshot 2024-09-24 at 17 23 01"
src="https://github.com/user-attachments/assets/85375897-504d-4e75-98e4-98e6a9671f98">
Example of payload when user is not logged in
```
{"hostName":"localhost",
"pathname":"/welcome",
"locale":"en-US",
"userAgent":"Mozilla/5.0",
"href":"http://localhost:3001/welcome",
"referrer":"",
"timeZone":"Europe/Barcelona"}
```
Example of payload when user is logged in
```
{"userId":"2020202",
"workspaceId":"202",
"workspaceDisplayName":"Apple",
"workspaceDomainName":"apple.dev",
"hostName":"localhost",
"pathname":"/objects/companies",
"locale":"en-US",
"userAgent":"Mozilla/5.0Chrome/128.0.0.0Safari/537.36",
"href":"http://localhost:3001/objects/companies",
"referrer":"",
"timeZone":"Europe/Paris"}
```
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
## Context
The api currently allows empty array in the IN filter but the expected
behaviour is not very clear. Typeorm seems to return all records when it
is empty which could lead to undesired result. Instead we decided to
throw an error.
I've updated the FE accordingly to skip calls when array is empty.
<img width="696" alt="Screenshot 2024-09-23 at 14 20 28"
src="https://github.com/user-attachments/assets/4b641430-ff17-40a6-bbc5-75e9a1d55f50">
### Summary
This PR introduces several integration tests, a mix of manually written
tests and those generated using the `generate-integration-tests` Python
script located in the `scripts` folder.
### Tests Added:
- **Authentication tests**: Validating login, registration, and token
handling.
- **FindMany queries**: Fetching multiple records for all existing
entities that do not require input arguments.
### How the Integration Tests Work:
- A `setupTest` function is called during the Jest test run. This
function initializes a test instance of the application and exposes it
on a dedicated port.
- Since tests are executed in isolated workers, they do not have direct
access to the in-memory app instance. Instead, the tests query the
application through the exposed port.
- A static accessToken is used, this one as a big expiration time so it
will never expire (365 years)
- The queries are executed, and the results are validated against
expected outcomes.
### Current State and Next Steps:
- These tests currently run using the existing development seed data. We
plan to introduce more comprehensive test data using `faker` to improve
coverage.
- At the moment, the only mutation tests implemented are for
authentication. Future updates should include broader mutation testing
for other entities.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
We had to remove soft-deletion on default filters due to the missing
indexes. We now generate composite indexes with the foreign key
containing the deletedAt column as well which should improve
performances
From PR: #6626Resolves#6763Resolves#6055Resolves#6782
## GTK
I retain the 'Invite by link' feature to prevent any breaking changes.
We could make the invitation by link optional through an admin setting,
allowing users to rely solely on personal invitations.
## Todo
- [x] Add an expiration date to an invitation
- [x] Allow to renew an invitation to postpone the expiration date
- [x] Refresh the UI
- [x] Add the new personal token in the link sent to new user
- [x] Display an error if a user tries to use an expired invitation
- [x] Display an error if a user uses another mail than the one in the
invitation
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Looks like insert() does not return foreign keys. We could eventually
call findMany after but it seems that's what save() is doing so I'm
replacing insert with save.
```typescript
/**
* Flag to determine whether the entity that is being persisted
* should be reloaded during the persistence operation.
*
* It will work only on databases which does not support RETURNING / OUTPUT statement.
* Enabled by default.
*/
reload?: boolean;
```
Note: save() also does an upsert by default with no way to configure
that so if we want to keep that behaviour we will need to add a check
before
```typescript
if (args.upsert) {
const existingRecords = await repository.findBy({
id: Any(args.data.map((record) => record.id)),
});
...
```
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
- avoid failing when missing cache (used for command)
- remove unused load cache function. Cache will be always re-created
when trying to fetch if not existing
## Context
This PR introduces createOne/createMany through the new graphql query
runner.
Trying to use twentyOrm wrapper as much as possible, in this case here
the args are already converted from "metadata-like" structure (including
composite fields) as graphql input to typeorm / raw columns (I had to
introduce a little fix there).
Keep in mind that I'm not using the new graphql query runner parsing
classes here, especially the selected-fields part, because typeorm
already returns all the record columns in the InsertResult object
(including default values such as id, createdAt, ...). That also means
relation objects will be returned as NULL in the gql response but we
don't handle nested creation for the moment so it should be fine.
Note: also removing the feature flag from findOne/findMany
In this PR:
- removing ugprade-0.24 commands as we are releasing 0.30
- introducing cache:flush command
- refactoring upgrade command and sync-metadata command to use the
ActiveWorkspacesCommand so they consistently run on all workspaces or
selected workspaces
Fixes:
- clear localStorage on sign out
- fix missing workspaceMember in verify resolver
- do not throw on datasource already destroyed exception which can
happen with race condition when several resolvers are resolving in
parallel
In this PR:
1. Refactor guards to avoid duplicated queries: WorkspaceAuthGuard and
UserAuthGuard only check for existence of workspace and user in the
request without querying the database
This https://github.com/twentyhq/twenty/pull/7006 introduced a
regression.
The goal was to set "isSoftDeletable" to all standard objects but it was
done at the wrong level, meaning it was setting the boolean correctly
but not creating the corresponding fieldMetadata.
I took the occasion to update the new graphql query runner to use that
boolean and automatically add a filter on soft delete in case it's true.
Also adding **IsQueryRunnerTwentyORMEnabled** by default in the seeds
- Create a workflow version when the user visits an empty workflow.
- If the trigger is not defined yet and the user selects either the
standard object type or the event type first, we automatically select
the first option of the other value. Indeed, every state update is
automatically saved on the backend and we need both standard object and
event types to save the event name.
- Introduces a change in the backend. I removed the assertions that
throw when a workflow version is not complete, that is, when it doesn't
have a defined trigger, which is the case when scaffolding a new
workflow with a first empty workflow version.
- We should keep validating the workflow versions, at least when we
publish them. That should be done in a second step.
