In the BE we throw custom errors with precise error codes (e.g.
"LABEL_ALREADY_EXISTS") before catching them in filters and rethrowing
BaseGraphQLErrors (standard errors such as NotFoundError, UserInputError
etc.).
In the FE we were grouping sentries based on the error codes but we were
actually grouping by very broad codes such as "NOT_FOUND" or
"BAD_USER_INPUT", extracted from the BaseGraphQLErrors.
To fix that, we update the BaseGraphQLError constructor api to allow to
pass on the CustomError directly and retrieve from it the original code
and store it in existing property `subCode` that we will use in the FE
to send errors to sentry.
This new api also eases usage of `userFriendlyMessage` that is passed on
to the api response and therefore to the FE when CustomError is passed
on directly to the BaseGraphQLError constructor.
# Introduction
- Added `INDEX` action generation
- Refactored the naming, mainly `WorkspaceMigrationV2ObjectInput` ->
`FlattenObjectMetadata` and same for field. The transpilation will be
done above, agnostically of the workspace migration
Still need to:
- Create testing toolbox and follow each testing pattern for each
actions and make a complex one ( remove static current tests )
- Handle standard and custom edges cases
Notes:
`workspace-migration-v2/types` and `workspace-migration-v2/utils` could
be located outside of this folder, my hunch is that we will move them
once we work on flatten tranpilers
Fixes https://github.com/twentyhq/twenty/issues/12867
Issue:
when you have a variable `toto` which is: `Record<string, MyType>` and
you do toto['xxx'], this will be typed as `MyType` instead of `MyType |
undefined`
Solutions:
- activate `noUncheckedIndexedAccess` check in tsconfig, this is the
preferred solution but will take time to get there (this raises 600+
errors)
- use a Map: cf https://github.com/twentyhq/twenty/pull/13125/files
- set the type to Partial<Record<string, MyType>>. Drawback is that when
you do Object.values(toto), you'll get `Array<MyType | undefined>`.
Hence why we have to filter these behind
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/d0a0bfed-c441-4e53-84c2-2da98ccbcf50"
/>
- update publishOneServerlessFunction so it does not break the workflow
if failing
- update upgrade documentation to update `docker-compose.yml` when
mutated
Fixes https://github.com/twentyhq/twenty/issues/13058
Large PR, sorry for that. Don't hesitate to reach me to have full
context (env. 500lines for integration and unit tests)
- Add connect logic in Workspace Entity Manager
- Update QueryDeepPartialEntity type to enable dev to use connect
- Add integration test on createOne / createMany
- Add unit test to cover main utils
- Remove feature flag on connect
closes https://github.com/twentyhq/core-team-issues/issues/1148
closes https://github.com/twentyhq/core-team-issues/issues/1147
I rebuilt the advanced filters used in views and workflow search for a
specific filter step.
Components structure remains the same, using `stepFilterGroups` and
`stepFilters`. But those filters are directly sent to backend.
Also re-using the same kind of states we use for advanced filters to
share the current filters used. And a context to share what's coming
from workflow props (function to update step settings and readonly)
⚠️ this PR only focusses on the content of the step. There is still a
lot to do on the filter icon behavior in the workflow
https://github.com/user-attachments/assets/8a6a76f0-11fa-444a-82b9-71fc96b18af4
In this PR
- introduction of fieldPermission entity
- addition of upsertFieldPermission in role resolver
- computing of permissions taking fieldPermission into account. In order
to limit what is stored in Redis we only store fields restrictions. For
instance for objectMetadata with id XXX with a restriction on field with
id YYY we store:
`"XXX":{"canRead":true,"canUpdate":false,"canSoftDelete":false,"canDestroy":false,"restrictedFields":{"YYY":{"canRead":false,"canUpdate":null}}}`
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
## Expected behavior
Described behavior regarding: (update | create) x (custom | standard) x
(icon, label, name, isSynced)
**Custom:**
- Field RELATION create: name, label, isSynced, icon should be editable
- Field RELATION update: name should not, icon label, isSynced should
- For other fields, icon, label, name, isSynced should be editable at
field creation | update
To simplify: Field RELATION name should not be editable at update
**Standards**
- Field: create does not makes sense
- Field: name should not, icon label, isSynced should (this will end up
in overrides)
To simplify, no Field RELATION edge case, name should not be editable at
update
**Note:** the FE logic is quite different as the UI is hiding some
details behind the syncWithLabel. See my comments and TODO there
## What I've tested:
(update | create) x (custom | standard) x (icon, label, name, isSynced,
description)
Updates yarn to the latest version 4.9.2 (from 4.4.0).
Also removes the explicit `enableHardenedMode` from yarnrc as it
significantly slows down installation.
This is already enabled automatically for pull requests on Github, thus
preventing lockfile poisoning where it's relevant.
See <https://yarnpkg.com/features/security#hardened-mode>:
> in most cases you won't even have to think about it - the hardened
mode is enabled by default when Yarn detects it runs in a pull request
from a public GitHub repository.
It can additionally be enabled explicitly for specific CI jobs by using
an environment variable, if desired:
> The hardened mode can be set (or disabled) [...] by defining
`YARN_ENABLE_HARDENED_MODE=1|0` in your environment variables
If this is the case, yarn still recommends **not** enabling it
everywhere:
> **DANGER**
>
> The hardened mode makes installs significantly slower as Yarn has to
query the registry to make sure the information contained in the
lockfile are accurate. If your CI pipeline runs multiple jobs, we
recommend disabling the hardened mode in all but one of them so as to
limit the performance impact.
---------
Co-authored-by: prastoin <paul@twenty.com>
In certain scenarios, the front directory may not be writable.
When this is the case, writing the patched `index.html` should be
skipped just like when the file does not exist.
For brevity, I have replaced the `existsSync` check with a try-catch
that catches any errors occuring during read
or write of the index file.
The alternative would be `fs.accessSync` with `W_OK`, but that would
still throw an error if the file is not writable so I think it is
reasonable to skip it altogether and go straight for the read and write
attempts.
A specific scenario where the front directory is immutable is NixOS,
where the directory may be located in the read-only nix store.
# Introduction
In this PR we've mainly refactor the typing to be extending existing
entities.
Also handling relations through the field abstraction layer rather than
a dedicated one. We reverted midway
We then still need to:
- Handle indexing
- Uniqueness
- Add strong coverage and avoid static inline snapshoting as right now +
building a coherent testing set
- Deprecate the `standardId` in favor of a `uniqueIdentifier` on each
`objectMetadata` and `fieldMetadata`
- Rename types `input` to `flattened`
- Handle custom or non custom edit edge cases. ( e.g cannot delete a
standard field or object )
## Notes
Right I preferred including too many information ( whole object and
field input ) in the action context, we might wanna evict redundant
information in the future when implementing the runners
This PR is purely technical, it does produces any functional change to
the user
- add Lock mecanism to run steps concurrently
- update `workflow-executor.workspace-service.ts` to handle multi branch
workflow execution
- stop passing `context` through steps, it causes race condition issue
- refactor a little bit
- simplify `workflow-run.workspace-service.ts` to prepare `output` and
`context` removal
- move workflowRun status computing from `run-workflow.job.ts` to
`workflow-executor.workspace-service.ts`
## NOTA BENE
When a code step depends of 2 parents like in this config (see image
below)
If the form is submitted before the "Code - 2s" step succeed, the branch
merge "Form" step is launched twice.
- once because form is submission Succeed resumes the workflow in an
asynchronous job
- the second time is when the asynchronous job is launched when "Code -
2s" is succeeded
- the merge "Form" step makes the workflow waiting for response to
trigger the resume in another job
- during that time, the first resume job is launched, running the merge
"Form" step again
This issue only occurs with branch workflows. It will be solved by
checking if the currentStepToExecute is already in a SUCCESS state or
not
<img width="505" alt="image"
src="https://github.com/user-attachments/assets/b73839a1-16fe-45e1-a0d9-3efa26ab4f8b"
/>
# Introduction
In this PR we've initialized the `workspace-migration-v2` folder.
Focusing on the builder in the first place.
From now it contains:
- Basic temporary types ( `fieldMetadataEntity` and
`ObjectMetadataEntity` )
- Object actions builder ( create, delete, update )
- Fields actions builder ( create, delete ) ( update coming in a
following PR )
We will still have to handle specific conditions such as:
- Index creation
- Uniqueness addition removal
- Relation
We also need to determine when we want to compute and transpile the
object no field `uniqueIdentifier`
We're aiming to merge this first in order to avoid accumulating code in
this PR
---------
Co-authored-by: prastoin <paul@twenty.com>
This PR is raised to close the issue #13044
But there are some doubts that needs to be approved.
If the fields are not custom then we were saving the changes in
**standardOverrides** obj in which only three fields are
overridableFields **label, icon & description** can be updated for a
field.
You can see this in _before-update-one-field.hook.ts_ on line 85
```ts
const overridableFields = ['label', 'icon', 'description'];
```
If the field to be updated are from these three we are putting this in a
**standardOverrides** obj and passing it
However in our _field-metadata.service.ts_ file. We have **updateOne**
function inside it we have wrote a condition if **isCustom** is false
then the purpose was to build the updatableFields from the
**standardOverrides** obj that we got in **fieldMetadataInput** but
there was an error in it. As you can see below
```ts
const updatableFieldInput =
existingFieldMetadata.isCustom === false
? this.buildUpdatableStandardFieldInput(
fieldMetadataInput,
existingFieldMetadata,
)
: fieldMetadataInput;
```
However, the issue was that we were placing the entire
**standardOverrides** object inside **updatableFieldInput** again —
instead of merging its individual fields (label, icon, description)
directly into the update payload.
This PR fixes that by correctly applying the overrides into the
top-level object.
Please refer to the file changes for the full context.
But the thing i don't know.
[ ] - Is this the correct expected flow??
[ ]- Will this change break anything elsewhere?
I still have doubts on these two. Let me know if I missed something.
---------
Co-authored-by: Jagss24 <btwitsjagannat12@gmail.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
In this PR, I'm fixing a bug introduced in recent performance work on
the cache.
Bug context: https://github.com/twentyhq/twenty/issues/12865
Related PR opened by a contributor:
https://github.com/twentyhq/twenty/pull/13003
## Root cause
We cache all objectMetadataItems at graphql level : see
`useCachedMetadata` hook:
- instead of going through the regular resolvers, we direlcty load data
from the cache. However this data must be localized regarding labels and
descriptions
In a precedent refactoring, we introduced the notion of locale in the
cache key. However, the user locale was not properly taken into account
as we did not have the information in this hook.
## Fix
1. **Introduce locale in userWorkspace entity**. The locale is stored on
workspaceMember in each postgres workspaceSchema (workspace_xxx) which
is the alter ego of userWorkspace in postgres core schema. Note that we
can't store it in user as a user can be part of multiple workspaces (the
locale already there must be seen as a default for this user), and we
cannot rely on workspaceMember as we would need to query the
workspaceSchema in the authentication layer which we want to avoid for
performance reasons.
2. During request hydration from token (containing the userWorkspaceId),
we fetch the userWorkspace and store it in the Request (this impact both
AuthContext and Request interface)
3. Leverage userWorkspace.locale in the useCachedMetadata hook
## Additional notes
There is no need to change the way we store and retrieve the
object-metadata-maps object itself which is different from the graphql
layer cache. object-metadadata-maps are not localized
Context :
- Phones import is a bit complex if not all subfields are provided.
- Phones subfield validation are absent or different from BE validation.
Solution :
- Normalize callingCode and countryCode validation (BE/FE)
- Ease phone import if only phoneNumber is provided
Currently, when a server query or mutation from the front-end fails, the
error message defined server-side is displayed in a snackbar in the
front-end.
These error messages usually contain technical details that don't belong
to the user interface, such as "ObjectMetadataCollection not found" or
"invalid ENUM value for ...".
**BE**
In addition to the original error message that is still needed (for the
request response, debugging, sentry monitoring etc.), we add a
`displayedErrorMessage` that will be used in the snackbars. It's only
relevant to add it for the messages that will reach the FE (ie. not in
jobs or in rest api for instance) and if it can help the user sort out /
fix things (ie. we do add displayedErrorMessage for "Cannot create
multiple draft versions for the same workflow" or "Cannot delete
[field], please update the label identifier field first", but not
"Object metadata does not exist"), even if in practice in the FE users
should not be able to perform an action that will not work (ie should
not be able to save creation of multiple draft versions of the same
workflows).
**FE**
To ease the usage we replaced enqueueSnackBar with enqueueErrorSnackBar
and enqueueSuccessSnackBar with an api that only requires to pass on the
error.
If no displayedErrorMessage is specified then the default error message
is `An error occured.`
When we use a record field in a form, record relations are displayed as
available variables in following step.
But those are actually empty at execution.
When choosing the record in the form and submitting, we enrich the
record id with the full record before starting the workflow again. But
we were not adding the relations to that enrichment.
This PR does not produce any functional change
First step of the workflow branch feature
- add gather `workflowRun.output` and `workflowRun.context` into one
column `workflowRun.runContext`
- add a command to fill `runContext` from `output` and `context` in
existing records
- maintain `runContext` up to date during workflow runs
Modifying the data-model can sometimes fail in the middle of your
operation, due to the way we handle both metadata update and schema
migration separately, a field can be created while the associated column
creation failed (same for object/table and such). This is also an issue
because WorkspaceMigrations are then stored as FAILED can never really
recovered by themselves so the schema is broken and we can't update the
models anymore.
This PR adds a executeMigrationFromPendingMigrationsWithinTransaction
method where we can (and must) pass a queryRunner executing a
transaction, which should come from the metadata services so that if
anything during metadata update OR schema update fails, it rolls back
everything (this also mean a workspaceMigration should never stay in a
failed state now).
This also fixes some issues with migration not running in the correct
order due to having the same timestamp and having to do some weird logic
to fix that.
This is a first step and fix before working on a much more reliable
solution in the upcoming weeks where we will refactor the way we
interact with the data model.
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
# Context
We had an error saying "Unknown error importing calendar events for
[...]: Access token is undefined or empty. Please provide a valid token.
For more help -
https://github.com/microsoftgraph/msgraph-sdk-javascript/blob/dev/docs/CustomAuthenticationProvider.md
"
Reason is that the access token method for microsoft is a bit different
than the one from google. And in microsoft case, we want to check the
access token in the authProvider in case it fails. Currently it was not
catched, so it broke services above that counted on the accesstoken to
be valid.
That ended in UNKNOWN failure for our calendar event fetch service.
# Solution
This PR should solve the issue since :
1. forcing the method to break if accesstoken renewal fails
2. logs will help to know what kind of errors will be sent in case we
need to tackle this issue again
3. we now throw TEMPORARY error instead of unknown, allowing 3
getClientConfig failure before it is definitive
Why so many changes while it should have been simple :
The root cause is the `authProvider` from
`'@microsoft/microsoft-graph-client'` npm package. It does not throw a
custom error, and we cannot catch it on calling `Client.init`. Errors
only occurs when the client from
```
const client = this.microsoftOAuth2ClientManagerService.getOAuth2Client(refreshtoken)
```
is used, as in `client.api('/messages').get().catch(err => [...])`
So we need to go in every call using the client and catch errors, and
rethrow whenver we need as a newly created message type
`MessageImportDriverExceptionCode.CLIENT_NOT_AVAILABLE`
We discussed 1. and 2. with @bosiraphael already
I added 3. to make our system more robust without waiting for more
failures
# Related
Fixes : https://github.com/twentyhq/twenty/issues/12880
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Previous logic was using the previous step output and filtering items
that were passing filters.
What we actually want is:
- send filters, right operand being always a step output key, left
operand being either a key, either a value
- resolve those filter variables
- apply the filters to decide whether the condition is passed or not
Fixes: #12722
The problem is that there is no TS_VECTOR field in workflow objects.
Thus, I have added this field to three objects: workflow,
workflowVersions, and workflowRuns.
---------
Co-authored-by: Thomas Trompette <thomas.trompette@sfr.fr>
RestApiExceptionFilter is used as an exception filter for the core
controller which is used for crud operations on our objects (equivalent
of our dynamic queries findManyPeople etc. on the graphql API).
Exceptions were leading a 400 / BadRequestException response status
which can be confusing to users.
By default we should actually throw a 500 if the error was not handled
priorily, but we have not implemented input validation for the REST api
so we fear to be flooded with errors that should not be 500 but 400 due
to user inputs. A solution should be brought [with this
ticket](https://github.com/twentyhq/core-team-issues/issues/1027) but it
has not been prioritized yet.
- new status `ENQUEUED` added. With a command to backfill
- counter in cache per workspace, managed by a new service
[workflow-run-queue.workspace-service.ts](https://github.com/twentyhq/twenty/compare/tt-improve-workflow-run-queueing?expand=1#diff-1e2de2a48cd482a3bd7e8dedf1150a19d0b200afbd9282181a24ecddddb56927)
- cron added that will run every minute to look for not started
workflows
Here is the new flow:
- When executing a workflow, we check if the queue is not full. If not,
run is created as `ENQUEUED` and the run workflow job is triggered as
usual. If full, create the run as NOT_STARTED and do not trigger the job
- Cron will look for NOT_STARTED workflows and queue some if there is
some place again
- Only MANUAL and Form submit skip the queue limit
