Commit Graph

163 Commits

Author SHA1 Message Date
ec4e4740d2 Make workflow statuses more resilient (#12263)
Workflow statuses are often broken. I did not figured out why yet. But I
see two causes that can be fixed:
- statuses calculation are really complicated today, just to spare a
call to the database
- job is not indempotent, it is using the combination of the previous
statuses + the update to calculate the new statuses. Which means that
once broken, next updates will be broken as well

Instead, we now:
- fetch workflow versions
- get the statuses from these.
It simplifies the code and make the job indempotent.
2025-05-26 13:05:18 +00:00
362d540aac Misc. of sentry improvements (#12233)
This PR mixes various initiatives to improve visibility on sentry 

**1. Catch errors on workflow jobs**
commit [catch workflowTriggerExceptions in job
handle](1dbba8c9e2)
@thomtrp 

**2. Fix type in messagingImportExceptionHandler** 
commit [fix type issue on
messagingImportExceptionHandler](919bb3844c)
@guillim 

**3. Catch invalid uuid errors thrown by Postgres by rightfully typing
expected id as uuid**
commits [use UUIDFilter instead of IDFilter to get graphqlError in case
of malformed
id](57cc315efe),
[use UUIDFilter
(2)](304553d770),
[fix ids typed as UUID instead of
ID](f95d6319cf)
@Weiko 
⚠️⚠️⚠️ when we deploy this PR we need to flush the schema types from
redis as this PR changes them ⚠️⚠️⚠️


**4. Do not group UNKNOWN errors together**
commit [do not group unknown errors
together](c299b39c8f)
Some CustomException classes have introduced UNKNOWN error codes as a
default fallback error code. We use CustomException codes to group
issues together, but we don't want to do it with UNKNOWN error as they
may not have anything in common. For exemple [this sentry for UNKNOWN
code](https://twenty-v7.sentry.io/issues/6605750776/events/a72272d8941b4fa2add9b1f39c196d3f/?environment=prod&environment=prod-eu&project=4507072499810304&query=Unknown&referrer=next-event&stream_index=0)
groups together "Unknown error importing calendar events for calendar
channel...", "Insufficent permissions...", to name a few.

**5. Improve postgres error grouping**
commit [group together postgres
errors](567c25495e)
Postgres error are thrown by typeORM as QueryFailedError. we have a lot
of them on sentry where they are badly grouped They are currently
grouped on sentry according to the stack trace, which leads them to
sometimes be grouped even if they don't have anything in common : for
exemple [this sentry for
QueryFailedError](https://twenty-v7.sentry.io/issues/6563624590/events/2d636821e27a448595b647b4b5a7d6a8/?environment=prod&environment=prod-eu&project=4507072499810304&query=is%3Aunresolved%20%21issue.type%3A%5Bperformance_consecutive_db_queries%2Cperformance_consecutive_http%2Cperformance_file_io_main_thread%2Cperformance_db_main_thread%2Cperformance_n_plus_one_db_queries%2Cperformance_n_plus_one_api_calls%2Cperformance_p95_endpoint_regression%2Cperformance_slow_db_query%2Cperformance_render_blocking_asset_span%2Cperformance_uncompressed_assets%2Cperformance_http_overhead%2Cperformance_large_http_payload%5D%20timesSeen%3A%3E10&referrer=previous-event&sort=date&stream_index=0)
groups together "user mapping not found for "postgres" and "invalide
type for uuid: 'fallback-id'" to name a few. I attempted to improve the
grouping by grouping them with a new custom fingerPrint composed of the
[code returned by
Postgres](https://www.postgresql.org/docs/current/errcodes-appendix.html)
+ the truncated operation name (Find, Aggregate, Check...). This is
still not ideal as postgres code are quite broad - we could have the
same error code for two Find operations with different causes. let's
give this a try !
2025-05-23 13:36:02 +00:00
ddab513c0b patch messages in thread (#12191)
closes https://github.com/twentyhq/twenty/issues/12190
2025-05-23 10:24:13 +00:00
4ac47c2a1b Improve logs in workflow trigger (#12215)
- distinguish logs coming from webhook and job triggers
- add workspace and workflow ids to help debugging

Hard to debug sentry issue:

https://twenty-v7.sentry.io/issues/6605607134/?project=4507072499810304&query=&referrer=issue-stream&stream_index=7
2025-05-22 14:56:30 +02:00
578f07374b 922 remove todos from 11809 after release (#12183)
- remove todos
- remove workflow-event-listener.workspace-entity.ts
2025-05-21 16:15:48 +02:00
5c16b2752d Fix workflow sub objects updates validation (#12169)
Fixes https://github.com/twentyhq/core-team-issues/issues/983

- Enabling workflow version name update, even if not in draft status
- Assert draft when updating steps
- Enabling workflow run name update
2025-05-21 13:11:32 +00:00
442f8dbe3c [QRQC_2] No implicitAny in twenty-server (#12075)
# Introduction
Following https://github.com/twentyhq/twenty/pull/12068
Related with https://github.com/twentyhq/core-team-issues/issues/975

We're enabling `noImplicitAny` handled few use case manually, added a
`ts-expect-error` to the others, we should plan to handle them in the
future
2025-05-15 18:23:22 +02:00
a8423e8503 [QRQC_2] No explicit any in twenty-server (#12068)
# Introduction

Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder

Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles

## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc

## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment

## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
    ignoreRestArgs?: boolean;
```

---------

Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
2025-05-15 16:26:38 +02:00
6a40ec604c Validate existing fields on creation (#12057)
Fixes https://github.com/twentyhq/twenty/issues/12040

When fields are deleted but still used in workflows we do not update
create record action settings.
It breaks all following workflow execution and the user cannot update
the settings anymore.

This PR fixes the bug by filtering on existing fields.

Next step will be to clean settings on field deletion. Adding it to fast
follows.

Also lowering throttle limit because some infinite loops are not
catched.
2025-05-15 13:24:57 +02:00
9c2b88870f Improve sentry grouping (#12007)
This PR attemps at improving sentry grouping and filtering by 
- Using the exceptionCode as the fingerprint when the error is a
customException. For this to work in this PR we are now throwing
customExceptions instead of internalServerError deprived of their code.
They will still be converted to Internal server errors when sent back as
response
- Filtering 4xx issues where it was missing (for emailVerification
because errors were not handled, for invalid captcha and billing errors
because they are httpErrors and not graphqlErrors)

---------

Co-authored-by: Félix Malfait <felix@twenty.com>
2025-05-14 09:00:06 +00:00
0202586d36 Fix message channel processing (#12021)
Several users have complained about not being able to read their emails
anymore.

This is because the find-messages post query hook is expecting
ObjectRecord[] as an input but is actually getting a graphql Connection

Typing was wrong. This PR fixes the typing and make sure the post query
hook always get an ObjectRecord[]
2025-05-13 21:16:23 +02:00
45d4845b26 Remove old relations (#11993)
This is a first PR to remove old relation logic

Next steps:
- remove relationMetadata from cache
- remove relationMetadata table content and structure
- refactor relationDefinition to leverage field.settings instead
2025-05-13 11:28:22 +02:00
e96afe444f Create filter action (#11904)
Figma
https://www.figma.com/design/xt8O9mFeLl46C5InWwoMrN/Twenty?node-id=59956-288587&t=Dkp83eigIgb3DO6W-11

Issue
https://github.com/orgs/twentyhq/projects/1/views/3?filterQuery=sprint%3A%40current+assignee%3A%40me&pane=issue&itemId=108202682&issue=twentyhq%7Ccore-team-issues%7C897

- filters will be stored as existing GQL filters. It will avoid
re-building frontend
- `applyFilter` function will take the filter and a JS array in input
and returns the filtered array
- filter action calls the util then returns an empty result and error if
no data in the output array. It will end the workflow gracefully.

Example of action:

```
{
      "id": "9d4aeee9-5b78-4053-9615-c367e901ed71",
      "name": "Filter",
      "type": "FILTER",
      "valid": false,
      "settings": {
        "input": {
          "filter": {
            "employees": {
              "gt": 300,
            },
          }
        }
      }
}
```
2025-05-07 14:52:03 +02:00
a9e73c6340 [permissions] Add permissions check layer in entityManager (#11818)
First and main step of
https://github.com/twentyhq/core-team-issues/issues/747

We are implementing a permission check layer in our custom
WorkspaceEntityManager by overriding all the db-executing methods (this
PR only overrides some as a POC, the rest will be done in the next PR).
Our custom repositories call entity managers under the hood to interact
with the db so this solves the repositories case too.
This is still behind the feature flag IsPermissionsV2Enabled.

In the next PR
- finish overriding all the methods required in WorkspaceEntityManager
- add tests
2025-05-05 14:06:54 +00:00
6128d660c2 Fix form record picker field (#11817)
- enrich response so the record is available in the step output. Today
this is available in the schema but only the id is set
- make the full record picker clickable instead of the arrow only

<img width="467" alt="Capture d’écran 2025-04-30 à 16 08 04"
src="https://github.com/user-attachments/assets/db74b9a6-7f1d-4e54-bf06-9be3d67ee398"
/>
2025-05-05 12:58:11 +00:00
49b7f5255f Update what is being audit logged (#11833)
No need to audit log workflow runs as it's already a form of audit log.
Add more audit log for other objects
Rename MessagingTelemetry to MessagingMonitoring
Merge Analytics and Audit in one (Audit)

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-05-04 14:35:41 +02:00
849a35955a 866 refactor cron trigger only one cron each minutes triggers all cron triggers (#11809)
<img width="1123" alt="image"
src="https://github.com/user-attachments/assets/75447922-81dd-4cfc-805d-f511f73cc778"
/>
2025-04-30 17:08:47 +02:00
d8b2e1fb34 Start using next step ids (#11683)
- update workflow executor
- update next step ids on step creation/deletion
- use these in workflow run
- use these in variables
2025-04-29 16:29:25 +02:00
4d7dbb1991 Fix missing on delete cascade for relation (#11725)
Follow up from https://github.com/twentyhq/twenty/pull/10217
Adding onDelete action on Many_to_one sides
2025-04-24 18:32:14 +02:00
4257f30f12 Permission checks on twentyORM global manager (#11477)
In this PR we are handling permissions when using
twentyORMGlobalManager,
and handling permissions for rest api and api key
2025-04-23 17:57:48 +02:00
28a1354928 Allow webhook only on active versions (#11705)
- webhook call should only be allow on active versions
- fixing exceptions that are wrongly sent to sentry
2025-04-23 17:28:41 +02:00
1333c64d98 Add diff in workflow update events + fix existing (#11700)
Closes https://github.com/twentyhq/twenty/issues/11676.

Diff should be available for objects displaying timeline activities. It
should also fit the existing type.
2025-04-23 15:43:41 +02:00
cc29c25176 feat: new relation sync-metadata, twenty-orm, create/update (#10217)
Fix
https://github.com/twentyhq/core-team-issues/issues/330#issue-2827026606
and
https://github.com/twentyhq/core-team-issues/issues/327#issue-2827001814

What this PR does when `isNewRelationEnabled` is set to `true`:
- [x] Drop the creation of the  foreign key as a `FieldMetadata`
- [x] Stop creating `RelationMetadata`
- [x] Properly fill `FieldMetadata` of type `RELATION` during the sync
command
- [x] Use new relation settings in TwentyORM
- [x] Properly create `FieldMetadata` relations when we create a new
object
- [x] Handle `database:reset` with new relations

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
2025-04-22 19:01:39 +02:00
33e0794da9 Update workflow statuses in real time (#11653)
Statuses are maintained by an async job. Those are calculations that we
would like to avoid using in both frontend and backend. Using push
updates are an easier way.



https://github.com/user-attachments/assets/31e44a82-08a8-4100-a38e-c965d5c73ee8
2025-04-22 11:03:12 +02:00
8112c3aab3 Fix workflow run executed by empty (#11642)
Workflow run executed by should not be empty when the workflow is
unamed. Adding Workflow as fallback

<img width="1072" alt="Capture d’écran 2025-04-17 à 19 17 12"
src="https://github.com/user-attachments/assets/e4cc3360-06e5-47e8-ba05-28a0eafab994"
/>
2025-04-18 09:26:51 +00:00
42e060ac74 Ws poc (#11293)
related to https://github.com/twentyhq/core-team-issues/issues/601

## Done
- add a `onDbEvent` `Subscription` graphql endpoint to listen to
database_event using what we have done with webhooks:
- you can subscribe to any `action` (created, updated, ...) for any
`objectNameSingular` or a specific `recordId`. Parameters are nullable
and treated as wildcards when null.
  - returns events with following shape
```typescript
  @Field(() => String)
  eventId: string;

  @Field()
  emittedAt: string;

  @Field(() => DatabaseEventAction)
  action: DatabaseEventAction;

  @Field(() => String)
  objectNameSingular: string;

  @Field(() => GraphQLJSON)
  record: ObjectRecord;

  @Field(() => [String], { nullable: true })
  updatedFields?: string[];
```
- front provide a componentEffect `<ListenRecordUpdatesEffect />` that
listen for an `objectNameSingular`, a `recordId` and a list of
`listenedFields`. It subscribes to record updates and updates its apollo
cached value for specified `listenedFields`
- subscription is protected with credentials

## Result

Here is an application with `workflowRun`


https://github.com/user-attachments/assets/c964d857-3b54-495f-bf14-587ba26c5a8c

---------

Co-authored-by: prastoin <paul@twenty.com>
2025-04-17 16:03:51 +02:00
78e10b2da5 Update next step ids on step update (#11605)
When inserting a new step between step 1 et step 2, then step 1 should
have the new step as next step id, add stop having step 2.

When deleting a step, we link the parent and next steps together. It may
change in the future
2025-04-16 15:30:05 +02:00
c40baf036c Add command to backfill next step id on version and run steps (#11579)
- add next step id on step
- backfill next step id on step, except for the last one
- backfill flow for workflow run, when it exists

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-04-15 15:26:58 +00:00
27f542e132 Use transformer in workflows (#11497)
We do not manage rich text properly in workflows. This is because API
has a layer called transformer service. Looks a bit as a duplicate of
format data, but this api layer was already there for position anyway.
Using it in workflow record actions.

I hope at some point we merged formatData util and transformer.
2025-04-11 16:31:45 +00:00
5b1091b19e Store trigger output (#11503)
As title. Trigger output is the workflow payload
2025-04-10 14:36:15 +02:00
bd3ec6d5e3 rename core-module environment to twenty-config (#11445)
closes https://github.com/twentyhq/core-team-issues/issues/759
2025-04-09 14:11:26 +02:00
f121c94d4a 701 workflow improve webhook triggers (#11455)
as title

Nota bene: I did not filter execution by http method. A POST webhook
trigger can be triggered by a GET request for more flexibility. Tell me
if you think it is a mistake


https://github.com/user-attachments/assets/1833cbea-51a8-4772-bcd8-088d6a087e79
2025-04-08 19:01:22 +00:00
752eb93836 disable workflow execution if billing issue (#11374)
closes https://github.com/twentyhq/core-team-issues/issues/404
2025-04-03 14:18:44 +00:00
a062a17229 Add form date field (#11360)
Builder
<img width="498" alt="Capture d’écran 2025-04-02 à 19 02 20"
src="https://github.com/user-attachments/assets/29db9fa9-aae4-4d1f-98f2-0b2371f944f1"
/>

Execution
<img width="837" alt="Capture d’écran 2025-04-02 à 19 02 47"
src="https://github.com/user-attachments/assets/ce1c442c-3c06-4f7e-99d6-3eb8fb1d2428"
/>
2025-04-03 10:10:43 +02:00
7488c6727a Add record picker in form action (#11331)
Record picker becomes a form field that could be used in another context
than workflows.

Settings
<img width="488" alt="Capture d’écran 2025-04-02 à 10 55 53"
src="https://github.com/user-attachments/assets/a9fc09ff-28cd-4ede-8aaa-af1e986cda8e"
/>

Execution
<img width="936" alt="Capture d’écran 2025-04-02 à 10 57 36"
src="https://github.com/user-attachments/assets/d796aeeb-cae1-4e59-b388-5b8d08739ea8"
/>
2025-04-02 17:08:33 +02:00
e74c8723d0 Add position in run / version + block creation from generated API (#11318)
- Add position during workflow version / creation. It will allow to have
the versions and runs ordered
- Block the creation from generated api for versions. We use workflow
post hooks or create from draft
2025-04-01 15:29:54 +02:00
023d071103 Set record position on workflow creation (#11308)
- Migrate record position factory to core-modules 
- set position on record creation
2025-04-01 09:50:43 +00:00
8d35454dd8 Form action fast follows (#11242)
- on form action, add trash on hover and fix spacing
- fix dark mode
- second batch of icon fixes

<img width="692" alt="Capture d’écran 2025-03-27 à 18 25 48"
src="https://github.com/user-attachments/assets/963f50d2-12ee-425c-97a3-2ad0c6bb5c57"
/>
2025-03-28 10:11:01 +01:00
049a065307 Update workflow run step (#11125)
Currently, when filling the form, values are not saved in the action
settings. This is an issue because we do not see the response in the
node settings, only in the output of the step.

This PR:
- adds a new endpoint to update a step in the run flow output
- updates this flow when a step is updated



https://github.com/user-attachments/assets/2e74a010-a0d2-4b87-bd1f-1c91f7ca6b60

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-03-24 17:42:15 +01:00
9ad8287dbc [REFACTOR] twenty-shared multi barrel and CJS/ESM build with preconstruct (#11083)
# Introduction

In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)

For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682

close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590

## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )

## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617

## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960

## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level

## Conclusion
As always any suggestions are welcomed !
2025-03-22 19:16:06 +01:00
c50cdd9510 Execute workflow form action (#11099)
- create a form filler component
- send the response on submit
- put back a field name. We need it for the step output
- validate a form is well set before activation

TODO:
- we need to refresh to see the form submitted. We need to discuss about
a strategy
- the response is not saved in the step settings. We need a new endpoint
to update workflow run step



https://github.com/user-attachments/assets/0f34a6cd-ed8c-4d9a-a1d4-51455cc83443
2025-03-21 17:38:14 +00:00
d99f027e8d 400 workflows webhooks trigger (#11041)
https://github.com/user-attachments/assets/dc0ece22-4d87-417f-b9e1-a11c3fd52ce8
2025-03-20 10:12:52 +00:00
5ba3c09b14 [CHORE] Prepare twenty-serverfor twenty-shared multi barrel refactor (#11022)
# Introduction
While running https://github.com/twentyhq/twenty/pull/10960 scripts
discovers few issues:
- Invalid named folder `pre-hooks.ts`
- Mock consuming outbound imported module resulting in consumed before
initialization
2025-03-19 11:26:42 +00:00
7921ee5216 Stop emitting update event if before===after (#11004)
as title
2025-03-19 10:49:21 +01:00
0ce91d84c1 Allow to add and delete fields (#10990)
- Allow to add a new field
- On field click, display a delete button
- Use id instead of names for fields



https://github.com/user-attachments/assets/4ebffe22-225a-4bae-aa49-99e66170181a
2025-03-18 17:24:52 +01:00
f4a362b53a Add base form action without logic (#10811)
<img width="1298" alt="Capture d’écran 2025-03-12 à 15 32 27"
src="https://github.com/user-attachments/assets/8a3140e5-e165-445e-a718-748aa76b525c"
/>
2025-03-12 17:05:31 +00:00
1b0413bf8b Add days schedule trigger (#10800)
<img width="1470" alt="image"
src="https://github.com/user-attachments/assets/660d79ba-60c7-4874-aa82-80a7575366ba"
/>
2025-03-12 15:25:07 +00:00
341439092a send email refacto (#10691)
working refacto with min module deps of messaging-import-manager-module
2025-03-06 12:56:40 +00:00
d151b1329c Generate fake form from metadata (#10641)
- add name to form field metadata
- extract field generation from object record schema
- use field generation to generate field from metadata
- add tests
2025-03-04 13:25:29 +01:00
cbd4a15148 Add test on workflow executor (#10589)
As title
2025-02-28 18:38:49 +01:00