- Improve the rest api by introducing startingAfter/endingBefore (we
previously had lastCursor), and moving pageInfo/totalCount outside of
the data object.
- Fix broken GraphQL playground on website
- Improve analytics by sending server url
- Removing existing listener that was backfilling created records
without position
- Switch to a job that backfill all objects within workspace
- Adapting `FIND_BY_POSITION` so it can fetch objects without position.
Currently we needed to input a number
- refactor record position factory and record position query factory
- override position if not present during createMany
To avoid overriding the same positions for all data in createMany, the
logic is:
- if inserted last, use last position + arg index + 1
- if inserted first, use first position - arg index - 1
In this PR, I'm refactoring the messaging module into smaller pieces
that have **ONE** responsibility: import messages, clean messages,
handle message participant creation, instead of having ~30 modules (1
per service, jobs, cron, ...). This is mandatory to start introducing
drivers (gmails, office365, ...) IMO. It is too difficult to enforce
common interfaces as we have too many interfaces (30 modules...). All
modules should not be exposed
Right now, we have services that are almost functions:
do-that-and-this.service.ts / do-that-and-this.module.ts
I believe we should have something more organized at a high level and it
does not matter that much if we have a bit of code duplicates.
Note that the proposal is not fully implemented in the current PR that
has only focused on messaging folder (biggest part)
Here is the high level proposal:
- connected-account: token-refresher
- blocklist
- messaging: message-importer, message-cleaner, message-participants,
... (right now I'm keeping a big messaging-common but this will
disappear see below)
- calendar: calendar-importer, calendar-cleaner, ...
Consequences:
1) It's OK to re-implement several times some things. Example:
- error handling in connected-account, messaging, and calendar instead
of trying to unify. They are actually different error handling. The only
things that might be in common is the GmailError => CommonError parsing
and I'm not even sure it makes a lot of sense as these 3 apis might have
different format actually
- auto-creation. Calendar and Messaging could actually have different
rules
2) **We should not have circular dependencies:**
- I believe this was the reason why we had so many modules, to be able
to cherry pick the one we wanted to avoid circular deps. This is not the
right approach IMO, we need architect the whole messaging by defining
high level blocks that won't have circular dependencies by design. If we
encounter one, we should rethink and break the block in a way that makes
sense.
- ex: connected-account.resolver is not in the same module as
token-refresher. ==> connected-account.resolver => message-importer (as
we trigger full sync job when we connect an account) => token-refresher
(as we refresh token on message import).
connected-account.resolver and token-refresher both in connected-account
folder but should be in different modules. Otherwise it's a circular
dependency. It does not mean that we should create 1 module per service
as it was done before
In a nutshell: The code needs to be thought in term of reponsibilities
and in a way that enforce high level interfaces (and avoid circular
dependencies)
Bonus: As you can see, this code is also removing a lot of code because
of the removal of many .module.ts (also because I'm removing the sync
scripts v2 feature flag end removing old code)
Bonus: I have prefixed services name with Messaging to improve dev xp.
GmailErrorHandler could be different between MessagingGmailErrorHandler
and CalendarGmailErrorHandler for instance
Query read timeouts happen when a remote server is not available. It
breaks:
- the remote server show page
- the record table page of imported remote tables
This PR will catch the exception so it does not go to Sentry in both
cases.
Also did 2 renaming.
For remotes, we will only create the foreign key, without the relation
metadata. Expected behavior will be:
- possible to create an activity. But the remote object will not be
displayed in the relations of the activity
- the remote objects should not be available in the search for relations
Also switched the number settings to an enum, since we now have to
handle `BigInt` case.
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
## Context
Since pg_graphql does not return specific error/exception, we have to
map the error message and throw validation errors when needed
This PR adds a check on unicity constraint error returned by pg_graphql
when we are trying to insert duplicate records and returns a 400 instead
of being handled by the exceptionHandler as a 500.
## Context
Yoga can catch its own errors and we don't want to convert them again.
Moreover those errors don't have an "originalError" property and should
be schema related only (400 validation) so we only want to send them
back to the API caller without going through the exception handler.
Also fixed an issue in the createMany which was throwing a 500 when id
was missing from the creation payload. It seems the FE is always sending
an ID but it should actually be optional since the DB can generate one.
This is a regression from the new UUID validation introduced a few weeks
ago.
## Context
JobsModule is hard to maintain because we provide all the jobs there,
including their dependencies. This PR aims to split jobs in dedicated
modules.
Various fixes
- Remote objects are read-only for now, we already hide and block most
of the write actions but the button that allows you to add a new record
in an empty collection was still visible.
- CreatedAt is not mandatory on remote objects (at least for now) so it
was breaking the show page, it now checks if createdAt exists and is not
null before trying to display the human readable format `Added x days
ago`
- The filters are overwritten in query-runner-args.factory.ts to handle
NUMBER field type, this was only working with filters like
```
{
"id": {
"in": [
1
]
}
```
but not with more depth such as
```
"and": [
{},
{
"id": {
"in": [
1
]
}
}
]
```
- Fixes CREATE FOREIGN TABLE raw query which was missing ",".
## Context
We have a non-nullable constraint on authorId in attachments and
documents, until we have soft-deletion we need to handle deletion of
workspace-members and their attachments/documents.
This PR introduces pre-hooks to deleteOne/deleteMany
This is called when a user deletes a workspace-member from the members
page
Next: needs to be done on user level as well. This is called when users
try to delete their own accounts. I've seen other issues such as
re-creating a user with a previously used email failing.
Adding stripe integration by making the server logic independent of the
input fields:
- query factories (remote server, foreign data wrapper, foreign table)
to loop on fields and values without hardcoding the names of the fields
- adding stripe input and type
- add the logic to handle static schema. Simply creating a big object to
store into the server
Additional work:
- rename username field to user. This is the input intended for postgres
user mapping and we now need a matching by name
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
## Context
#4774
## How was it tested
Locally
## In further PRs
- Update connection status upon page change
- Adapt Info banner to dark mode
- placeholders for form
## Context
@lucasbordeau introduced a new Yoga plugin that allows us to cache our
requests (👏), see https://github.com/twentyhq/twenty/pull/5189
I'm simply updating the implementation to allow us to use different
cache storage types such as redis
Also adding a check so it does not use cache for other operations than
ObjectMetadataItems
## Test
locally, first call takes 340ms, 2nd takes 30ms with 'redis' and 13ms
with 'memory'
In this PR I'm introducing a simple custom graphql-yoga plugin to create
a caching mechanism specific to our metadata.
The cache key is made of : workspace id + workspace cache version, with
this the cache is automatically invalidated each time a change is made
on the workspace metadata.
New strategy:
- add settings field on FieldMetadata. Contains a boolean isIdField and
for numbers, a precision
- if idField, the graphql scalar returned will be a GraphQL id. This
will allow the app to work even for ids that are not uuid
- remove globals dateScalar and numberScalar modes. These were not used
- set limit as Integer
- check manually in query runner mutations that we send a valid id
Todo left:
- remove WorkspaceBuildSchemaOptions since this is not used anymore.
Will do in another PR
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
Co-authored-by: Weiko <corentin@twenty.com>
In this PR:
- Follow up on #5170 as we did not take into account not logged in users
- only apply throttler on root fields to avoid performance overhead
Refactored the code to introduce two different concepts:
- AuditLogs (immutable, raw data)
- TimelineActivities (user-friendly, transformed data)
Still some work needed:
- Add message, files, calendar events to timeline (~2 hours if done
naively)
- Refactor repository to try to abstract concept when we can (tbd, wait
for Twenty ORM)
- Introduce ability to display child timelines on parent timeline with
filtering (~2 days)
- Improve UI: add links to open note/task, improve diff display, etc
(half a day)
- Decide the path forward for Task vs Notes: either introduce a new
field type "Record Type" and start going into that direction ; or split
in two objects?
- Trigger updates when a field is changed (will be solved by real-time /
websockets: 2 weeks)
- Integrate behavioral events (1 day for POC, 1 week for
clean/documented)
<img width="1248" alt="Screenshot 2024-04-12 at 09 24 49"
src="https://github.com/twentyhq/twenty/assets/6399865/9428db1a-ab2b-492c-8b0b-d4d9a36e81fa">
Experiment using swc instead of tsc (as we did the switch on
twenty-front)
It's **much** faster (at least 5x) but has stricter requirements.
I fixed the build but there's still an error while starting the server,
opening this PR for discussion.
Checkout the branch and try `nx build:swc twenty-server`
Read: https://docs.nestjs.com/recipes/swc#common-pitfalls
Added isAuditLogged column to object-metadata-entity.ts
This is my first open source pull request. Please do let me know if made
any mistake. I will be greatfull. Thank u
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
- Implemented dataloader package on metadata graphql server
- Implemented a dataloader for relation metadata module
---------
Co-authored-by: Jérémy M <jeremy.magrin@gmail.com>
We have discovered that GraphQL inputs for fields of type ids in create
/ update input where using a more permissive ID type than the type used
in FilterInput in queries.
This PRs fixes it and make sure that all Input are using UUID graphql
scalar types
This PR is dropping the column `targetColumnMap` of fieldMetadata
entities.
The goal of this column was to properly map field to their respecting
column in the table.
We decide to drop it and instead compute the column name on the fly when
we need it, as it's more easier to support.
Some parts of the code has been refactored to try making implementation
of composite type more easier to understand and maintain.
Fix#3760
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Backend: Adding a new util function that throw an error if the
objectMetadata is remote
Frontend: hiding the save button when remote
Also renaming `useObjectMetadataItemForSettings` since this hook is used
in other places than settings and is not in the settings repo. Name can
definitely be challenged!
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
Foreign table id cannot be a foreign key of a base table. But the
current code use foreign keys to link object metadata with activities,
events... So we will:
- create a column without creating a foreign key
- add a comment on the table schema so pg_graphql sees it as a foreign
key
This PR:
- refactor a bit object metadata service so the mutation creation is
separated into an util
- adds the mutation creation for remote object relations
- add a new type of mutation to create a comment
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
When writing to the normalized cache (record), it's crucial to use _refs
for relationships to avoid many problems. Essentially, we only deal with
level 0 and generate all fields to be comfortable with their defaults.
When writing in queries (which should be very rare, the only cases are
prefetch and the case of activities due to the nested query; I've
reduced this to a single file for activities
usePrepareFindManyActivitiesQuery 🙂), it's important to use queryFields
to avoid bugs. I've implemented them on the side of query generation and
record generation.
When doing an updateOne / createOne, etc., it's necessary to distinguish
between optimistic writing (which we actually want to do with _refs) and
the server response without refs. This allows for a clean write in the
optimistic cache without worrying about nesting (as the first point).
To simplify the whole activities part, write to the normalized cache
first. Then, base queries on it in an idempotent manner. This way,
there's no need to worry about the current page or action. The
normalized cache is up-to-date, so I update the queries. Same idea as
for optimisticEffects, actually.
Finally, I've triggered optimisticEffects rather than the manual update
of many queries.
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
