## Why
After the changes on relations, index on relations were skipped by the
syncmetadata service, so no more migrations were generated for relation
fields.
We wanted to fix this.
## Test
This PR adds unit tests for the `createIndexMigration` utility in the
workspace migration builder. The tests cover:
- Creating index migrations for simple fields (e.g., text fields)
- Creating index migrations for relation fields (ensuring correct column
naming, e.g., `authorId` for the `author` objectmetadataname)
## Excluded
The delete index on relation does not need the column names so i don't
think i needed to work on this method. I might be wrong.
## Checklist
- [x] Added/updated unit tests for index migration creation
- [x] Verified correct handling of simple and relation fields
- [x] Ensured all tests pass
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
# Indexes
### TLDR:
Putting indexes back, except relation ones
### Details:
- Added index synchronization logic back (it was removed previously in
45d4845b26)
in the sync-metadata service.
- for unique inedexes, a command will create unicity again by handling
duplicates that were cretated since the
45d4845b26
was triggered
Closes https://github.com/twentyhq/core-team-issues/issues/748
In the frame of the work on permissions we
- remove all raw queries possible to use repositories instead
- forbid usage workspaceDataSource.executeRawQueries()
- restrict usage of workspaceDataSource.query() to force developers to
pass on shouldBypassPermissionChecks to use it.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
# Introduction
Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder
Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles
## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc
## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment
## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
ignoreRestArgs?: boolean;
```
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
This is a first PR to remove old relation logic
Next steps:
- remove relationMetadata from cache
- remove relationMetadata table content and structure
- refactor relationDefinition to leverage field.settings instead
## Context
While deploying the IS_NEW_RELATION_ENABLED (we don't compute relation
based on relationMetadata anymore) to existing workspace, I've tested to
run a sync-metadata post feature flag activation. This has raised two
issues:
- the workspaceMigration generator (which is over-complex and should be
refactored later) for fieldMetadata of type RELATION was not handling
settings update properly ;
- we need to delete existing fieldMetadata corresponding to the UUID
foreignKey as they are not needed anymore. This is handled as a 0.53
upgrade command as 0.53 will also come with the full removal of the old
relation system
---------
Co-authored-by: Etienne <45695613+etiennejouan@users.noreply.github.com>
Co-authored-by: prastoin <paul@twenty.com>
No need to audit log workflow runs as it's already a form of audit log.
Add more audit log for other objects
Rename MessagingTelemetry to MessagingMonitoring
Merge Analytics and Audit in one (Audit)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
In this PR:
- this should fix the sync metadata for new relation system
This goes with the recent PR:
https://github.com/twentyhq/twenty/pull/11725
What we want:
- ONE_TO_MANY relations should have no joinColumn and no onDelete
- MANY_TO_ONE should have both
This PR implements what's missing to have sub-field filtering.
There is a backend modification to save subFieldName, we just add this
field on view filter workspace entity.
This PR adds subFieldName where missing in frontend, notably in
applyFilter calls, that will be refactored soon.
Also fixes a bug in ViewBar where Add Filter button was at the right
side of the ViewBar, while it should be right after the chips section.
Another bug fixed where we wouldn't delete an empty record filter on
dropdown click outside from the view bar, which was already the case
where using the filter chip dropdown.
<img width="512" alt="image"
src="https://github.com/user-attachments/assets/e9a2f8d2-a66f-4800-853a-4df5c6b627a9"
/>
<img width="495" alt="image"
src="https://github.com/user-attachments/assets/7542697b-0689-4095-9c3c-b5e47875355f"
/>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
## Context
Workspace creation and more specifically sync-metadata performances are
bad at the moment. We are trying to identify bottlenecks and one of the
root causes is the migration runner that can take up to 10s when setting
up a new workspaces with all its tables.
First observation is we do a lot of things sequentially, mostly to make
the code easier to read and debug but it impacts performances. For
example, a table creation is done in two steps, we first ask typeorm to
create the table then ask typeorm to create columns (and sometimes
columns one by one), each instruction can take time because typeorm
seems to do some checks internally.
The proposition here is to try to merge migrations when possible, for
example when we create a table we want the migration to also contain the
columns it will contain so we can ask typeorm to add the columns at the
same time. We are also using batch operations when possible (addColumns
instead of addColumn, dropColumns instead of dropColumn)
Still, we could go further with foreign keys creations or/and try with
raw query directly.
## Test
New workspace creation:
See screenshot, 9865.40233296156ms is on main, the rest is after the
changes:
<img width="610" alt="Screenshot 2025-02-28 at 09 27 21"
src="https://github.com/user-attachments/assets/42e880ff-279e-4170-b705-009e4b72045c"
/>
ResetDB and Sync-metadata on an existing workspace commands still work
Simplifying a lot the upgrade system.
New way to upgrade:
`yarn command:prod upgrade`
New way to write upgrade commands (all wrapping is done for you)
```
override async runOnWorkspace({
index,
total,
workspaceId,
options,
}: RunOnWorkspaceArgs): Promise<void> {}
```
Also cleaning CommandModule imports to make it lighter
# Introduction
Historically we've been programmatically running sync metadata just
after all upgrade command's migration.
Adding back this behavior as default to the new dynamic modules
Duplicated already existing synchronize metadata logic as a quick fix as
we're about to iterate over commands next sprint
Workspace Member will get their own record page in the future.
This PR lays backend changes to prepare for this:
- Settings most fields on WorkspaceMember as system fields
- Renaming workspaceMember/workspaceMemberId to
forWorkspaceMember/forWorkspaceMemberId as it conflicts with the morph
relationship, if we want to be able to add a workspace member as
favorite
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
In this PR:
- adding logs to track workspace creation performance
- refactor useIsWorkspaceSuspended to be more generic
- only fetch favorites and views if workspace is Active to avoid error
messages on sign up (workspace is not created yet)
Proposal:
- Add a method in ActiveWorkspaceCommand to loop over workspace safely
(add counter, add try / catch, provide datasource with fresh cache,
destroy datasource => as we do always do it)
Also in this PR:
- make sure we clear all dataSources (and not only the one on metadata
version in RAM)
Closes https://github.com/twentyhq/core-team-issues/issues/410
- Added `openRecordIn` column in the `view` entity, which is set to
`SIDE_PANEL` by default
- Created a new option inside the view option dropdown to be able to set
`openRecordIn`
- Updated all record show page openings to reflect the setting behavior
- For `workflow`, `workflowVersion` and `workflowRun` (what I call
workflow objects), we want the default view `openRecordIn` to be set to
`RECORD_PAGE`. When seeding the views for the new workspaces, we set
`openRecordIn` to `RECORD_PAGE` for workflow objects. Since the workflow
objects views `openRecordIn` will be set to the default value
`SIDE_PANEL` for the existing workspaces when the sync metadata runs, I
created a script to run in the 0.43 update to update this value.
- Updated `closeCommandMenu` because of problems introduced by the
animate presence wrapper around the command menu. We now reset the
states at the end of the animation.
Note: We want to be able to open all workflow objects pages in the side
panel, but this requires some refactoring of the workflow module. For
now @Bonapara wanted to allow the possibility to change the
`openRecordIn` setting to `SIDE_PANEL` even for the workflows even if
it's buggy and not ready for the moment. Since this is an experimental
feature, it shouldn't cause too many problems.
While making sure the upgrade-0.42 command was working as expected to
upgrade from 0.41 to 0.42, we've detected that the standardId of the new
bodyV2 (type RICH_TEXT_V2) was not properly set ; standardId was not
correct.
This was forcing the sync-metadata command to try to re-create the field
again.
## Context
All objects have '...duplicates' resolver but only companies and people
have duplicate criteria (hard coded constant).
Gql schema and resolver should be created only if duplicate criteria
exist.
## Solution
- Add a new @WorkspaceDuplicateCriteria decorator at object level,
defining duplicate criteria for given object.
- Add a new duplicate criteria field in ObjectMetadata table
- Update schema and resolver building logic
- Update front requests for duplicate check (only for object with
criteria defined)
closes https://github.com/twentyhq/twenty/issues/9828
Fixes#9827
Also uncovered a conflict with `@objectType('Relation')` and
`@objectType('relation)`
I don't want to address it in this PR so I will create a followup issue
when we close this but I think there's a confusion between
Relation/RelationMetadata, it's unclear what is what
---------
Co-authored-by: Antoine Moreaux <moreaux.antoine@gmail.com>
## Context
<img width="1349" alt="Screenshot 2025-01-13 at 17 18 24"
src="https://github.com/user-attachments/assets/4f5da0e9-0245-41c6-bde2-4d52e0ba34ed"
/>
Feature flags are stored in DB and then cast as FeatureFlag gql type
from its corresponding enum.
This means if a value from the DB does not match that enum type, the gql
server will reject the call when returning the object in the resolver.
(see screenshot above)
To solve that, we want to do 2 things:
- The ORM should still return the feature flag even if it's not valid,
this is actually in the DB so we don't want to "hide" that, however we
now have a warning message.
- The service is not changed for the same reason, the limitation comes
from gql behaviour so this is not the goal of the service nor the ORM to
act on it (except the warning message)
- The resolver should be updated, here we want to filter-out non-valid
feature flags so it does not break the API.
Because featureFlags used to be auto-generated by nestjsquery and we
want to change its behavior, I had to manually create a resolveField for
featureFlags and remove the auto-generated one. That means we lose some
features such as filter/sort coming from nestjs-query pagination (which
is something we will want to implement once we will remove nestjs-query
but that's a whole other subject)
## Context
Following this https://github.com/twentyhq/twenty/issues/4199
This has not been fully implemented, after 5months of dead code I'm
removing the feature for the time being until we re-prioritise the
feature (unlikely during these next 6 months) to keep the codebase a bit
cleaner (no need to maintain dead features)
Feel free to reopen / revert this PR once feature is ready
## Test
locally after importing emails
Closes#9187
This pull request introduces a new feature and several enhancements for
managing webhook security by adding a secret field and enabling HMAC
signature-based authentication. Below is a detailed breakdown of the
changes made:
## Frontend Updates
### Secret Field on Webhook Edit Page
- Added a new **Secret** section on the webhook edit page.
- Includes a text input field for entering a webhook secret.
- Added a descriptive note explaining the purpose of the secret for
webhook authentication.
### State Management and Persistence
- Integrated the secret field into the Webhook type definition and state
management.
- Connected the secret field UI to the data layer, ensuring seamless
persistence of the secret field.
### Validation Improvement
- Trims leading and trailing whitespace from webhook secret inputs to
avoid potential validation issues.
## Backend Updates
### Database and Entity Changes
- Introduced a nullable `secret` field to the `WebhookWorkspaceEntity`
for securely storing webhook signing secrets.
- Field uses a standard field ID:
`20202020-97ce-410f-bff9-e9ccb038fb67`.
### Signature Generation
- Implemented HMAC-SHA256 signature generation for webhook payloads when
a secret is present:
- Signatures are added as a custom `X-Twenty-Webhook-Signature` header.
- Secret is excluded from the payload to maintain security.
### Enhanced Security Measures
- Added additional headers for enhanced security:
- **Timestamp Header**: Prevents replay attacks.
- **Nonce Header**: Mitigates duplicate requests.
- Updated the OpenAPI specification to include documentation on these
security-related headers and signature verification.
## Documentation Updates
- Updated OpenAPI documentation for webhook endpoints:
- Described security-related headers (signature, timestamp, nonce).
- Included detailed instructions for verifying HMAC signatures to assist
consumers.
## Testing and Demonstration
- [Loom Video
Link](https://www.loom.com/share/bd827e4d045f46d99f3c8186e5e5676a?sid=a5e61904-0536-4e82-8055-3d05e4598393):
Demonstrating the functionality of the secret field and webhook security
features.
- [Script Example
Link](https://runkit.com/samyakpiya/676af044040c0400086d400a): A script
showing how consumers can verify webhook authenticity using the HMAC
signature.
- [Testing Site
Instance](https://webhook.site/#!/view/3472468b-ebcd-4b7f-a083-c4ba20825bb4/6885fdce-8843-4d3f-8fe0-1d8abdd53f68/1):
Contains the logged requests sent during testing and is available for
review.
## Steps for Review
1. Verify the secret field functionality on the webhook edit page,
including state persistence and UI updates.
2. Review the security enhancements, including header additions and HMAC
signature generation.
3. Validate OpenAPI documentation changes for completeness and clarity.
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
Restoring https://github.com/twentyhq/twenty/pull/9185
Also fixing sync-metadata with test values in jsonb
## Test
sync-metadata on existing workspaces should replace colorSchema in both
metadata and workspaceMember tables
Fixes https://github.com/twentyhq/twenty/issues/8810
Fixes https://github.com/twentyhq/twenty/issues/5268
Fixes https://github.com/twentyhq/twenty/issues/8971
- Fixing Task/Note creation not sending position during creation
- Adding a command to backfill position being null, using existing
backfill command.
- Removed unused backfill job.
- Updated workspace entities to set position non-nullable and set a
default value to make it non-required on the API
- Updated position factory to set a default position for all objects
having a POSITION field instead of only company/people
- Moved the try/catch in each resolver factory calling
GraphqlQueryRunnerException handler, makes more sense to call it in the
actual graphql-query-runner and removing some duplicate codes
- Adding validations for input in QueryRunnerArgs factories
- Allow sync-metadata to override and sync defaultValues for certain
field types (that can't be updated by users)
- Removing health-check from sync-metadata command during force mode to
improve performances
Search vector fields based on `RICH_TEXT` fields were generated using a
treatment on `RICH_TEXT` fields's `body`
column, to only extract and index the core text.
([PR](https://github.com/twentyhq/twenty/pull/7953))
Actually our RICH_TEXT fields are of datatype `text` in our database,
allowing users to insert non-json values, which breaks the search vector
generation (as it expects json values).
Our vision is unclear for now: for instance we may want to turn
RICH_TEXT into a composite field where the plain text would be stored in
a different column.
So for now, we will (1) treat rich_text data as text, and (2) update the
search vector expressions for the existing workspaces.
