## Context
Regression on custom index creation where indexFieldMetadatas were not
saved properly in the DB. This is because we recently changed save() to
upsert() in the indexMetadataService and upsert does not handle nesting
insert properly.
I'm suggesting another fix where we separate indexMetadata creation and
index migration creation in 2 different functions. Since the goal was to
be able to recreate the index after being deleted when we changed the
tsvector expression and indexMetadata was actually not deleted, we
didn't need to recreate that part (hence the upsert) and only needed to
run a migration to create the actual index in the workspace schema.
I've updated the different services and now only call
createIndexMigration when we update a search vector expression.
Note: this is also fixing the sync-metadata command when running on a
workspace with a custom object (including the seeded workspace which has
the 'rocket' custom object), failing due to the missing 'searchVector'
indexFieldMetadata
## Context
This PR removes pg_graphql from the setup. It also updates the local
setup documentation accordingly.
Note: We removed local setup scripts to align with redis installation,
the setup should be much simpler since we don't rely on pg_graphql
anymore.
## Test
tested locally with docker + mac (brew) setup
## Context
WorkspaceIsUnique decorator allows us to create unique indexes on our
tables. Here Emails is a composite field containing a TEXT subfield. Due
to the fact that TEXT fields are non-nullable and have empty strings as
default values, adding a unique index on a TEXT field or any composite
containing TEXT as subfields will fail and throw if we try to create
more than 1 record that does not specify a value to the TEXT field.
This PR simply removes the index for the time being until we find a
solution
## Context
This PR removes workspace-query-runner/builder in preparation for fully
deprecating pg_graphql
next steps: Remove from the setup and make a command to remove comments
on schema/tables related to pg_graphql
## Context
This can be thrown when a type is not properly supported by
isRecordMatchingFilter. This should not happen but for some workspaces
with legacy types this should help debugging.
## Context
Because we can't create GIN indexes on primitive types and in this case
both tsvector and deletedAt (date), we had to create an index on the
tsvector only and had to make sure it wouldn't return results for
deleted records.
To handle this, we added a CASE in the generated expression to generate
an empty value if the row has been soft-deleted. This is a bit too
complex and won't allow search on softDeleted records in the future.
What we want to do is to make sure deleted records are not returned by
default by adding a WHERE clause in the search and still keep good
performances by also adding a BTREE index on deletedAt column.
When this was implemented, soft-deleted was not handled properly but now
typeorm query builder exclude soft deleted records by default which
means the WHERE clause is not necessary. As for the BTREE index on
deletedAt, this should be part on a broader effort to add it to all
tables and can be added in a later PR.
This PR simply updates the complex tsVectorExpression to only return the
expression regardless of the deletedAt column. Search won't return
soft-deleted records as explained above and perfs will be improved in an
upcoming version.
## Test
tested locally with reset db + created a new workspace. Since the
feature is not launched yet no more effort should be made.
<img width="1289" alt="Screenshot 2024-10-10 at 15 16 16"
src="https://github.com/user-attachments/assets/d9a6f06b-d738-4f1d-8053-2fa93bd8b4f8">
## Note
Some issues with optimistic rendering, we need to refresh once a record
has been deleted otherwise it will still be returned by the search
Following this https://github.com/twentyhq/twenty/pull/7428 we now need
to fix existing workspaces thanks to this migration command.
This command will fetch all standard objects and their fields then
filter out tables that don't have that column OR objects that have an
existing fieldMetadata "name" of type TEXT and delete the rest.
## Context
Name shouldn't be added to all tables, especially standard objects
because they already have their own labelIdentifierFieldMetadata
specified in the workspace-entity schema. This PR removes this column
from the "base" list of columns to add when creating a new object/table
and moves it to the object-metadata service that is, as of today, only
used for custom objects. Also had to modify the migration-runner to
handle column creation in a table creation migration (this was available
in the migration definition already but was not doing anything)
This also fixes an issue in standard objects that already have a "name"
field defined with a different field type, this is even more important
when the said field is a composite field. For example people already has
a FULL_NAME name field which clashes with the default TEXT name field
meaning it was only creating 1 field metadata for 'name' but 3 columns
were created: `name, nameFirstName, nameLastName`. This inconsistency
with metadata (which is our source of truth everywhere) brought some
issues (lately, converting back typeorm response to gql (including
composition) was broken).
Fixes https://github.com/twentyhq/twenty/issues/6859
This PR adds all the remaining resolvers for
- updateOne/updateMany
- createOne/createMany
- deleteOne/deleteMany
- destroyOne
- restoreMany
Also
- refactored the graphql-query-runner to be able to add other resolvers
without too much boilerplate.
- add missing events that were not sent anymore as well as webhooks
- make resolver injectable so they can inject other services as well
- use objectMetadataMap from cache instead of computing it multiple time
- various fixes (mutation not correctly parsing JSON, relationHelper
fetching data with empty ids set, ...)
Next steps:
- Wrapping query builder to handle DB events properly
- Move webhook emitters to db event listener
- Add pagination where it's missing (findDuplicates, nested relations,
etc...)
## Context
Now that we have improved performances, we can increase the export
feature page size from 30 to 200 (and probably above if results are
good). This should be ok since we are only querying the first level of
an object and omit relations.
I've moved this value to a constant.
## Context
The api currently allows empty array in the IN filter but the expected
behaviour is not very clear. Typeorm seems to return all records when it
is empty which could lead to undesired result. Instead we decided to
throw an error.
I've updated the FE accordingly to skip calls when array is empty.
<img width="696" alt="Screenshot 2024-09-23 at 14 20 28"
src="https://github.com/user-attachments/assets/4b641430-ff17-40a6-bbc5-75e9a1d55f50">
We had to remove soft-deletion on default filters due to the missing
indexes. We now generate composite indexes with the foreign key
containing the deletedAt column as well which should improve
performances
Looks like insert() does not return foreign keys. We could eventually
call findMany after but it seems that's what save() is doing so I'm
replacing insert with save.
```typescript
/**
* Flag to determine whether the entity that is being persisted
* should be reloaded during the persistence operation.
*
* It will work only on databases which does not support RETURNING / OUTPUT statement.
* Enabled by default.
*/
reload?: boolean;
```
Note: save() also does an upsert by default with no way to configure
that so if we want to keep that behaviour we will need to add a check
before
```typescript
if (args.upsert) {
const existingRecords = await repository.findBy({
id: Any(args.data.map((record) => record.id)),
});
...
```
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Context
We currently have a race condition when dealing with datasource
creation. This happen when multiple queries arrive at the same time (for
example graphql dataloaders) and the datasource is not created yet.
Since the datasource is stored in memory this can happen more often as
well and they were all triggering the datasource creation at the same
time.
I'm trying to fix the issue with promise memoization. Now, instead of
caching the datasource only, we also want to cache the promise of the
datasource creation and make the creation itself synchronous.
More info about promise memoization in this article for example:
https://www.jonmellman.com/posts/promise-memoization
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
This command was supposed to set all custom objects as softDeletable.
After some discussion we realised this bool was not used as intended so
we are removing it all together until we find a better usage (remote
objects for example). This PR removes the command which won't be needed
anymore
## Context
This PR introduces createOne/createMany through the new graphql query
runner.
Trying to use twentyOrm wrapper as much as possible, in this case here
the args are already converted from "metadata-like" structure (including
composite fields) as graphql input to typeorm / raw columns (I had to
introduce a little fix there).
Keep in mind that I'm not using the new graphql query runner parsing
classes here, especially the selected-fields part, because typeorm
already returns all the record columns in the InsertResult object
(including default values such as id, createdAt, ...). That also means
relation objects will be returned as NULL in the gql response but we
don't handle nested creation for the moment so it should be fine.
Note: also removing the feature flag from findOne/findMany
This https://github.com/twentyhq/twenty/pull/7006 introduced a
regression.
The goal was to set "isSoftDeletable" to all standard objects but it was
done at the wrong level, meaning it was setting the boolean correctly
but not creating the corresponding fieldMetadata.
I took the occasion to update the new graphql query runner to use that
boolean and automatically add a filter on soft delete in case it's true.
Also adding **IsQueryRunnerTwentyORMEnabled** by default in the seeds
## Context
The goal is to replace pg_graphql with our own ORM wrapper (TwentyORM).
This PR tries to add some parsing logic to convert graphql requests to
send to the ORM to replace pg_graphql implementation.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Adding more logs with Typeorm QueryFailedError in sync-metadata command
Example with a unicity constraint violation, to identify which column is
affected
<img width="841" alt="Screenshot 2024-08-09 at 14 56 05"
src="https://github.com/user-attachments/assets/c47fbb1d-77ee-4d7a-87e7-dbe54a6aa941">
In this case, this should help self-hosting users to know which key is a
duplicate during syncs after a version upgrade for example
## Context
UploadFile now returns the file token which we don't want when we save
the attachment in the DB due to its non-persistence so now the FE
removes the token query param before saving the attachment. This is most
likely not the right way to do it, we will need to refactor this part
before.
Also made sure we don't save the token in the DB for rich_text as well
and remove it before saving.
## Context
We recently introduced the new twenty ORM and used it in the update
methods in the query runner.
Initially we were using pg_graphql to fetch the record before updating
it allowing us to compare the before and the after and create a diff.
This diff is then used for the timeline activity creation. Now,
twentyORM is doing the fetch and pg_graphql is still doing the update
and their responses are not exactly the same, which means the diff is
not working as intended (e.g date types were always in the diff due to
one being in Date format and the other as a string)
This PR introduces a updatedFields property to the update event which
comes from the input. This is not ideal as this won't work for API users
that send the whole payload but will be sufficient enough for our FE
that only sends modified fields. We then compare only those fields in
the diff.
## Context
An object should always have a labelIdentifier (would be its primary key
at least). If the associated field is deleted by a user, it will break
the app. Ideally we should handle that on the DB level but we don't have
a FK for this column yet.
In the meantime I'm adding the validation check in the backend, note
that this is already handle on the FE side since the "archive/delete"
buttons don't appear for such fields so you need to reassign it to
another field first which is the desired behaviour.
## Context
LabelIdentifier and ImageIdentifier are metadata info attached to
objectMetadata that are used to display a record in a more readable way.
Those columns point to existing fields that are part of the object.
For example, for a relation picker of a person, we will show a record
using the "name" labelIdentifier and the "avatarUrl" imageIdentifier.
<img width="215" alt="Screenshot 2024-07-11 at 18 45 51"
src="https://github.com/twentyhq/twenty/assets/1834158/488f8294-0d7c-4209-b763-2499716ef29d">
Currently, the FE has a specific logic for company and people objects
and we have a way to update this value via the API for custom objects,
but the code is not flexible enough to change other standard objects.
This PR updates the WorkspaceEntity API so we can now provide the
labelIdentifier and imageIdentifier in the WorkspaceEntity decorator.
Example:
```typescript
@WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.activity,
namePlural: 'activities',
labelSingular: 'Activity',
labelPlural: 'Activities',
description: 'An activity',
icon: 'IconCheckbox',
labelIdentifierStandardId: ACTIVITY_STANDARD_FIELD_IDS.title,
})
@WorkspaceIsSystem()
export class ActivityWorkspaceEntity extends BaseWorkspaceEntity {
@WorkspaceField({
standardId: ACTIVITY_STANDARD_FIELD_IDS.title,
type: FieldMetadataType.TEXT,
label: 'Title',
description: 'Activity title',
icon: 'IconNotes',
})
title: string;
...
```
## Context
We've created a yoga (gql server) hook that catches requests and cache
them when needed. In practice we use it on the "objects" query because
this is often queried on the FE and it should never return something
different unless the schema has been intentionally changed by the user
when editing their data model (updating objects, fields, etc).
The issue here is we always cache the response regardless of its result,
even when it fails. This PR fixes that behaviour by only caching the
query response if it is successful.
I'm also fixing the cache key because the signature let users put
multiple operations and the cache key was not taking this into account
(we always use it on only one operation but we might have issues in the
future because another operation response could have erased the cached
response of another). Now the cache key contains the name of the
operation as well.
## Test
tested locally by manually throwing an error in the JWT auth guard
## Context
- Adding RATING sort and filter capabilities.
- Fixing isEmpty/isNotEmpty filters
- Fixing combined view filters so it combines filters per field metadata
and not per filter id. This is more a product question but to me it does
not make sense to apply multiples filters on the same field IF the
operations is wrapped in a AND. If at some point we want to put a OR
instead then that would make more sense
We call convertExceptionToGraphQLError in the exception handler for http
exceptions but we don't take into account those that already are
graphqlErrors and because of that the logic of convertExceptionToGraphql
is to fallback to a 500.
Now if the exception is a BaseGraphqlError (custom graphql error we
throw in the code), we throw them directly.
BEFORE
<img width="957" alt="Screenshot 2024-07-12 at 15 33 03"
src="https://github.com/user-attachments/assets/22ddae13-4996-4ad3-8f86-dd17c2922ca8">
AFTER
<img width="923" alt="Screenshot 2024-07-12 at 15 32 01"
src="https://github.com/user-attachments/assets/d3d6db93-6d28-495c-a4b4-ba4e47d45abd">
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Add a new command to delete objectMetadataId fieldMetadata that have a
wrong standard-id. This is because we have fixed the missing
objectMetadataId column but one already exists in the fieldMetadataId
with the wrong table. We should run this command before run the
sync-metadata.
Introduced a new module and command to run all the command associated
with the upgrade to 0.22. Not exactly sure with this structure but
ideally we would like to have only 1 command for version upgrades so
this is a first step.
## Context
We want to add an index on our foreign keys since PG does not do it for
us. An index can sometimes be expensive and not always meaningful
depending on different usages but in our case we decided to apply an
index for every foreign keys.
```typescript
@WorkspaceIndex()
@WorkspaceJoinColumn('author')
authorId: string;
```
This syntax is valid but since we want to apply it to every join column
I've decided to update the code of WorkspaceJoinColumn so it properly
registers a new index at the same time which is less error-prone.
Note: We had a bug on index name generation since postgres index names
are unique per schema and not table, the object metadata id (hashed) has
been added to the formula that generates the name of the index
## Test
Sync metadata. We have 45 join columns as of today per workspace, we
should see 45 rows inside IndexMetadata table
Insert inside AuditLog table are all failing due to objectMetadataId
column missing.
The FieldMetadata was sharing the same standard-id with another one
(objectName) so it was skipped during the comparison step of the
sync-metadata.
Running a sync-metadata again should fix this issue. Note that this
column is non-nullable so if the table contains existing records, it
will fail. However, since the insert was failing I'm assuming the table
is empty anyway.
## Context
Rating fields were not editable on the show page and kanban view when
they were null, this is because we don't have a way to leave the empty
state for fields that are editModeContentOnly.
~~This is actually an issue for bool fields (which is the other field
type that has editModeContentOnly) as well but they have default values
can't go be edited to NULL so it's not visible.~~
Actually let's fix bool, this could happen too
Hovering over "Empty" will now show the RatingField edit mode.
I'm not 100% sure about this solution though, we could also make this
behaviour on click? I preferred over since this is the behaviour on the
table view 🤔
## Test
https://github.com/twentyhq/twenty/assets/1834158/6825b5c3-2c62-41f2-8e03-343bc0e895e2
## Context
Our Flexible Schema engine dynamically generates entities/tables/APIs
for us but was not flexible enough to build indexes in the DB. With more
and more features involving heavy queries such as Messaging, we are now
adding a new WorkspaceIndex() decorator for our standard objects (will
come later for custom objects). This decorator will give enough
information to the workspace sync metadata manager to generate the
proper migrations that will create or drop indexes on demand.
To be aligned with the rest of the engine, we are adding 2 new tables:
IndexMetadata and IndexFieldMetadata, that will store the info of our
indexes.
## Implementation
```typescript
@WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.person,
namePlural: 'people',
labelSingular: 'Person',
labelPlural: 'People',
description: 'A person',
icon: 'IconUser',
})
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
@WorkspaceField({
standardId: PERSON_STANDARD_FIELD_IDS.email,
type: FieldMetadataType.EMAIL,
label: 'Email',
description: 'Contact’s Email',
icon: 'IconMail',
})
@WorkspaceIndex()
email: string;
```
By simply adding the WorkspaceIndex decorator, sync-metadata command
will create a new index for that column.
We can also add composite indexes, note that the order is important for
PSQL.
```typescript
@WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.person,
namePlural: 'people',
labelSingular: 'Person',
labelPlural: 'People',
description: 'A person',
icon: 'IconUser',
})
@WorkspaceIndex(['phone', 'email'])
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```
Currently composite fields and relation fields are not handled by
@WorkspaceIndex() and you will need to use this notation instead
```typescript
@WorkspaceIndex(['companyId', 'nameFirstName'])
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```
<img width="700" alt="Screenshot 2024-06-21 at 15 15 45"
src="https://github.com/twentyhq/twenty/assets/1834158/ac6da1d9-d315-40a4-9ba6-6ab9ae4709d4">
Next step: We might need to implement more complex index expressions,
this is why we have an expression column in IndexMetadata.
What I had in mind for the decorator, still open to discussion
```typescript
@WorkspaceIndex(['nameFirstName', 'nameLastName'], { expression: "$1 || ' ' || $2"})
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Context
Fixing `setIsFocused is not a function` and the fact that edit buttons
were not showing up anymore.
A new FieldFocusContextProvider has been introduced and added to
RecordInlineCell but not ActivityTargetsInlineCell. This should fix the
issue.
<img width="523" alt="Screenshot 2024-06-05 at 17 42 07"
src="https://github.com/twentyhq/twenty/assets/1834158/1c1f919e-3829-4e40-b573-3b1b75b7c16f">
Our exception handler has to filter out some errors/exceptions so they
are not caught by the ExceptionHandlerDriver (Logged or Sentry for
example). This is done for Http errors in the range of 4xx and also
makes sure they are converted back to Graphql validation errors.
However, graphql validation errors that are already managed by Yoga
(with Schema validation) should also be filtered out, this PR should fix
that behaviour
## Context
Since pg_graphql does not return specific error/exception, we have to
map the error message and throw validation errors when needed
This PR adds a check on unicity constraint error returned by pg_graphql
when we are trying to insert duplicate records and returns a 400 instead
of being handled by the exceptionHandler as a 500.
## Context
Yoga can catch its own errors and we don't want to convert them again.
Moreover those errors don't have an "originalError" property and should
be schema related only (400 validation) so we only want to send them
back to the API caller without going through the exception handler.
Also fixed an issue in the createMany which was throwing a 500 when id
was missing from the creation payload. It seems the FE is always sending
an ID but it should actually be optional since the DB can generate one.
This is a regression from the new UUID validation introduced a few weeks
ago.
## Context
JobsModule is hard to maintain because we provide all the jobs there,
including their dependencies. This PR aims to split jobs in dedicated
modules.
## Context
Because creating an object in metadata also generates a graphql type and
because graphql does not allow 2 types with the same name, we have to
manage a list of reserved keywords that can't be used as object names.
Currently we were maintaining a list of the core objects but we also
have to introduce composite fields that are also generated as gql types.
## Context
Positions are used within a view to display and sort the different
records of standard/custom object.
When we add a new record and want to put it before the existing first
record, we have to use float values to insert them in the DB and respect
the desired order. We are adding a new command that can be executed to
flatten those positions.
---------
Co-authored-by: bosiraphael <raphael.bosi@gmail.com>
## Context
There is no calendarChannel syncStatus column compared to the
messageChannel table. In the meantime, we are trying to infer its status
based on the fact that the connection hasn't failed and the sync is
enabled
Various fixes
- Remote objects are read-only for now, we already hide and block most
of the write actions but the button that allows you to add a new record
in an empty collection was still visible.
- CreatedAt is not mandatory on remote objects (at least for now) so it
was breaking the show page, it now checks if createdAt exists and is not
null before trying to display the human readable format `Added x days
ago`
- The filters are overwritten in query-runner-args.factory.ts to handle
NUMBER field type, this was only working with filters like
```
{
"id": {
"in": [
1
]
}
```
but not with more depth such as
```
"and": [
{},
{
"id": {
"in": [
1
]
}
}
]
```
- Fixes CREATE FOREIGN TABLE raw query which was missing ",".
## Context
We recently introduced a change that now throws a 401 if the token is
invalid or expired.
The first implementation is using an allow list and 'IntrospectionQuery'
was missing so the playground was broken.
The check has been updated and we now only check the excludedOperations
list if a token is not present. This is because some operations can be
both used as loggedIn and loggedOut so we want to validate the token for
those sometimes (and set the workspace, user, cache version, etc). Still
not a very clean solution imho.
## Context
We have a non-nullable constraint on authorId in attachments and
documents, until we have soft-deletion we need to handle deletion of
workspace-members and their attachments/documents.
This PR introduces pre-hooks to deleteOne/deleteMany
This is called when a user deletes a workspace-member from the members
page
Next: needs to be done on user level as well. This is called when users
try to delete their own accounts. I've seen other issues such as
re-creating a user with a previously used email failing.
## Context
Those settings are not implemented yet, we would like to move them to a
different page as well.
In the meantime, we are hiding them since we plan to launch calendar in
the next release and this won't be implemented before.
We will implement it in this
https://github.com/twentyhq/twenty/issues/5140
## Context
Currently, this middleware validates the token and stores the user,
workspace and cacheversion in the request object.
It only does so when a token is provided and ignores the middleware
logic if not. If the token is invalid or expired, the exception is
swallowed.
This PR removes the try/catch and adds an allowlist to skip the token
validation for operations executed while not signed-in.
I don't know a better way to do that with Nestjs. We can't easily add
the middleware per resolver without refactoring the flexible schema
engine so I'm doing it the other way around.
Fixes https://github.com/twentyhq/twenty/issues/5224
## Context
Currently we have an unicity constraint in the DB but we don't return a
clear error to the frontend before reaching the DB (which then throws a
500). This PR adds a validation check similar to what we have with field
creation
Now that we have persistent cache for schemas, we want to be able to
reset its state when users run the database:reset db otherwise schemas
won't be synced with the new DB state.
Note: In an upcoming PR, we want to be able to invalidate the cache on a
workspace level when we change the metadata schema through twenty
version upgrade
Weiko