In this PR, I'm fixing a bug introduced in recent performance work on
the cache.
Bug context: https://github.com/twentyhq/twenty/issues/12865
Related PR opened by a contributor:
https://github.com/twentyhq/twenty/pull/13003
## Root cause
We cache all objectMetadataItems at graphql level : see
`useCachedMetadata` hook:
- instead of going through the regular resolvers, we direlcty load data
from the cache. However this data must be localized regarding labels and
descriptions
In a precedent refactoring, we introduced the notion of locale in the
cache key. However, the user locale was not properly taken into account
as we did not have the information in this hook.
## Fix
1. **Introduce locale in userWorkspace entity**. The locale is stored on
workspaceMember in each postgres workspaceSchema (workspace_xxx) which
is the alter ego of userWorkspace in postgres core schema. Note that we
can't store it in user as a user can be part of multiple workspaces (the
locale already there must be seen as a default for this user), and we
cannot rely on workspaceMember as we would need to query the
workspaceSchema in the authentication layer which we want to avoid for
performance reasons.
2. During request hydration from token (containing the userWorkspaceId),
we fetch the userWorkspace and store it in the Request (this impact both
AuthContext and Request interface)
3. Leverage userWorkspace.locale in the useCachedMetadata hook
## Additional notes
There is no need to change the way we store and retrieve the
object-metadata-maps object itself which is different from the graphql
layer cache. object-metadadata-maps are not localized
Closes https://github.com/twentyhq/core-team-issues/issues/393
- enforcing object-records permission checks in resolvers for now. we
will move the logic to a lower level asap
- add integration tests that will still be useful when we have moved the
logic
- introduce guest seeded role to test limited permissions on
object-records
In this PR
- closing https://github.com/twentyhq/core-team-issues/issues/313
- adding permission gates on workspace settings and security settings
- adding integration tests for each of the protected setting and
security
# This PR
- Addressing #3644
- Migrates the `DELETE /rest/*` endpoint to use TwentyORM
- Factorizes common middleware logic into a common module
---------
Co-authored-by: martmull <martmull@hotmail.fr>
### Summary
This PR introduces several integration tests, a mix of manually written
tests and those generated using the `generate-integration-tests` Python
script located in the `scripts` folder.
### Tests Added:
- **Authentication tests**: Validating login, registration, and token
handling.
- **FindMany queries**: Fetching multiple records for all existing
entities that do not require input arguments.
### How the Integration Tests Work:
- A `setupTest` function is called during the Jest test run. This
function initializes a test instance of the application and exposes it
on a dedicated port.
- Since tests are executed in isolated workers, they do not have direct
access to the in-memory app instance. Instead, the tests query the
application through the exposed port.
- A static accessToken is used, this one as a big expiration time so it
will never expire (365 years)
- The queries are executed, and the results are validated against
expected outcomes.
### Current State and Next Steps:
- These tests currently run using the existing development seed data. We
plan to introduce more comprehensive test data using `faker` to improve
coverage.
- At the moment, the only mutation tests implemented are for
authentication. Future updates should include broader mutation testing
for other entities.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Fix https://github.com/twentyhq/twenty/issues/6669
- create a commun function `startWorkflowRun` that both create the run
object and the job for executing the workflow
- use it in both the `workflowEventJob` and the `runWorkflowVersion`
endpoint
Bonus:
- use filtering for exceptions instead of a util. It avoids doing a try
catch in all endpoint
## Context
As we grow, the messaging scripts are experiencing performance issues
forcing us to temporarily disable them on the cloud.
While investigating the performance, I have noticed that generating the
entity schema (for twentyORM) in the repository is taking ~500ms locally
on my Mac M2 so likely more on pods. Caching the entitySchema then!
I'm also clarifying naming around schemaVersion and cacheVersions ==>
both are renamed workspaceMetadataVersion and migrated to the workspace
table (the workspaceCacheVersion table is dropped).
This pull request introduces a new `FieldMetadataType` called `ACTOR`.
The primary objective of this new type is to add an extra column to the
following objects: `person`, `company`, `opportunity`, `note`, `task`,
and all custom objects.
This composite type contains three properties:
- `source`
```typescript
export enum FieldActorSource {
EMAIL = 'EMAIL',
CALENDAR = 'CALENDAR',
API = 'API',
IMPORT = 'IMPORT',
MANUAL = 'MANUAL',
}
```
- `workspaceMemberId`
- This property can be `undefined` in some cases and refers to the
member who created the record.
- `name`
- Serves as a fallback if the `workspaceMember` is deleted and is used
for other source types like `API`.
### Functionality
The pre-hook system has been updated to allow real-time argument
updates. When a record is created, a pre-hook can now compute and update
the arguments accordingly. This enhancement enables the `createdBy`
field to be populated with the correct values based on the
`authContext`.
The `authContext` now includes:
- An optional User entity
- An optional ApiKey entity
- The workspace entity
This provides access to the necessary data for the `createdBy` field.
In the GraphQL API, only the `source` can be specified in the
`createdBy` input. This allows the front-end to specify the source when
creating records from a CSV file.
### Front-End Handling
On the front-end, `orderBy` and `filter` are only applied to the name
property of the `ACTOR` composite type. Currently, we are unable to
apply these operations to the workspace member relation. This means that
if a workspace member changes their first name or last name, there may
be a mismatch because the name will differ from the new one. The name
displayed on the screen is based on the workspace member entity when
available.
### Missing Components
Currently, this PR does not include a `createdBy` value for the `MAIL`
and `CALENDAR` sources. These records are created in a job, and at
present, we only have access to the workspaceId within the job. To
address this, we should use a function similar to
`loadServiceWithContext`, which was recently removed from `TwentyORM`.
This function would allow us to pass the `authContext` to the jobs
without disrupting existing jobs.
Another PR will be created to handle these cases.
### Related Issues
Fixes issue #5155.
### Additional Notes
This PR doesn't include the migrations of the current records and views.
Everything works properly when the database is reset but this part is
still missing for now. We'll add that in another PR.
- There is a minor issue: front-end tests are broken since this commit:
[80c0fc7ff1).
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
### Overview
This PR introduces significant enhancements to the MessageQueue module
by integrating `@Processor`, `@Process`, and `@InjectMessageQueue`
decorators. These changes streamline the process of defining and
managing queue processors and job handlers, and also allow for
request-scoped handlers, improving compatibility with services that rely
on scoped providers like TwentyORM repositories.
### Key Features
1. **Decorator-based Job Handling**: Use `@Processor` and `@Process`
decorators to define job handlers declaratively.
2. **Request Scope Support**: Job handlers can be scoped per request,
enhancing integration with request-scoped services.
### Usage
#### Defining Processors and Job Handlers
The `@Processor` decorator is used to define a class that processes jobs
for a specific queue. The `@Process` decorator is applied to methods
within this class to define specific job handlers.
##### Example 1: Specific Job Handlers
```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';
@Processor('taskQueue')
export class TaskProcessor {
@Process('taskA')
async handleTaskA(job: { id: string, data: any }) {
console.log(`Handling task A with data:`, job.data);
// Logic for task A
}
@Process('taskB')
async handleTaskB(job: { id: string, data: any }) {
console.log(`Handling task B with data:`, job.data);
// Logic for task B
}
}
```
In the example above, `TaskProcessor` is responsible for processing jobs
in the `taskQueue`. The `handleTaskA` method will only be called for
jobs with the name `taskA`, while `handleTaskB` will be called for
`taskB` jobs.
##### Example 2: General Job Handler
```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';
@Processor('generalQueue')
export class GeneralProcessor {
@Process()
async handleAnyJob(job: { id: string, name: string, data: any }) {
console.log(`Handling job ${job.name} with data:`, job.data);
// Logic for any job
}
}
```
In this example, `GeneralProcessor` handles all jobs in the
`generalQueue`, regardless of the job name. The `handleAnyJob` method
will be invoked for every job added to the `generalQueue`.
#### Adding Jobs to a Queue
You can use the `@InjectMessageQueue` decorator to inject a queue into a
service and add jobs to it.
##### Example:
```typescript
import { Injectable } from '@nestjs/common';
import { InjectMessageQueue, MessageQueue } from 'src/engine/integrations/message-queue';
@Injectable()
export class TaskService {
constructor(
@InjectMessageQueue('taskQueue') private readonly taskQueue: MessageQueue,
) {}
async addTaskA(data: any) {
await this.taskQueue.add('taskA', data);
}
async addTaskB(data: any) {
await this.taskQueue.add('taskB', data);
}
}
```
In this example, `TaskService` adds jobs to the `taskQueue`. The
`addTaskA` and `addTaskB` methods add jobs named `taskA` and `taskB`,
respectively, to the queue.
#### Using Scoped Job Handlers
To utilize request-scoped job handlers, specify the scope in the
`@Processor` decorator. This is particularly useful for services that
use scoped repositories like those in TwentyORM.
##### Example:
```typescript
import { Processor, Process, InjectMessageQueue, Scope } from 'src/engine/integrations/message-queue';
@Processor({ name: 'scopedQueue', scope: Scope.REQUEST })
export class ScopedTaskProcessor {
@Process('scopedTask')
async handleScopedTask(job: { id: string, data: any }) {
console.log(`Handling scoped task with data:`, job.data);
// Logic for scoped task, which might use request-scoped services
}
}
```
Here, the `ScopedTaskProcessor` is associated with `scopedQueue` and
operates with request scope. This setup is essential when the job
handler relies on services that need to be instantiated per request,
such as scoped repositories.
### Migration Notes
- **Decorators**: Refactor job handlers to use `@Processor` and
`@Process` decorators.
- **Request Scope**: Utilize the scope option in `@Processor` if your
job handlers depend on request-scoped services.
Fix#5628
---------
Co-authored-by: Weiko <corentin@twenty.com>
Experiment using swc instead of tsc (as we did the switch on
twenty-front)
It's **much** faster (at least 5x) but has stricter requirements.
I fixed the build but there's still an error while starting the server,
opening this PR for discussion.
Checkout the branch and try `nx build:swc twenty-server`
Read: https://docs.nestjs.com/recipes/swc#common-pitfalls
