Commit Graph

43 Commits

Author SHA1 Message Date
ee6180a76f [Fix] Prevent fields name conflicts with composite subfields names (#6713)
At field creation we are checking the availability of the name by
comparing it to the other fields' names' on the object; but for
composite fields the fields' names' as indicated in the repository do
not exactly match the column names' on the tables (e.g "createdBy" field
is actually represented by columns createdByName, createdBySource etc.).

In this PR we prevent the conflict with the standard composite fields'
names.
There is still room for errors with the custom composite fields: for
example a custom composite field "address" of type address on a custom
object "listing" will introduce the columns addressAddressStreet1,
addressAddressStreet2 etc. while we won't prevent the user from later
creating a custom field named "addressAddressStreet1".
For now I decided not to tackle this as this seem extremely edgy + would
impact performance on creation of all fields while never actually useful
(I think).
2024-08-23 13:24:10 +02:00
7ca091faa5 Add isInactive to FieldMetadata decorator (#6623)
This PR was created by [GitStart](https://gitstart.com/) to address the
requirements from this ticket:
[TWNTY-4145](https://clients.gitstart.com/twenty/5449/tickets/TWNTY-4145).
This ticket was imported from:
[TWNTY-4145](https://github.com/twentyhq/twenty/issues/4145)

 --- 
### Description
This PR introduces the @isInActive() decorator to the standard field
metadata. \
- This gives the ability to allow workspaces to be able to be created
with standard fields as inactive *Helps prevent polluting existing
workspaces
- A new standard field can be added to* eg company-workspace-entity.ts
and the @WorkspaceIsInActive() decorator can be added to create it in
deactivated mode
sync-metadata command: `yarn command:prod workspace:sync-metadata -f`

### Refs
#4145 

### Demo

<https://www.loom.com/share/10c13e2614d749809cfe2d2d847e963e?sid=017bbfcf-53c6-4205-8ffc-8a09c416220a>\

<https://www.loom.com/share/6ab86bd36f344c999cb8dacdb82c7bb0?sid=13ac78a2-de44-4772-bb54-7b57975e7360>\

Fixes: #4145

---------

Co-authored-by: gitstart-twenty <gitstart-twenty@users.noreply.github.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
2024-08-22 18:37:06 +02:00
612a875c7b Remove performance logs (#6709)
We have found the root cause of the issue:
- when using a datasource (including the cached ones), we are fetching
ObjectMetadataCollection from cache (700kB). Datasource usage is
happening any time we are using twentyORM, which is everywhere in the
jobs and in some resolvers (including the GetCurrentUser one). This is
leading to a high load on redis and leading to the performance issues we
are seeing.
- we actually don't need to fetch this objectMetadataCollection while
using a cached datasource, only when we instantiate a new one
2024-08-21 14:17:11 +02:00
614a81860f Add logging on currentWorkspaceMember query (#6706)
We are experiencing slow GetCurrentUser endpoint, this is helping us
troubleshoot
2024-08-21 11:48:20 +02:00
17a1760afd Improve performance twenty orm (#6691)
## Context

As we grow, the messaging scripts are experiencing performance issues
forcing us to temporarily disable them on the cloud.
While investigating the performance, I have noticed that generating the
entity schema (for twentyORM) in the repository is taking ~500ms locally
on my Mac M2 so likely more on pods. Caching the entitySchema then!

I'm also clarifying naming around schemaVersion and cacheVersions ==>
both are renamed workspaceMetadataVersion and migrated to the workspace
table (the workspaceCacheVersion table is dropped).
2024-08-20 19:42:02 +02:00
db54469c8a feat: soft delete (#6576)
Implement soft delete on standards and custom objects.
This is a temporary solution, when we drop `pg_graphql` we should rely
on the `softDelete` functions of TypeORM.

---------

Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
2024-08-16 21:20:02 +02:00
40bbee8d9f 5x Fix cache performance issues (#6616)
Calling `getObjectMetadata` from `WorkspaceCacheStorageService` in every
query was causing big performance issues. The `objectMetadataCollection`
is now part of the `WorkspaceInternalContext` so we only instance it
once in the `WorkspaceDatasourceFactory`.
Queries are now much faster, for instance for TimelineCalendar, it went
from ~450ms to 80ms.
2024-08-13 17:54:55 +02:00
d5350e11a3 Remove some dead code (#6611)
We could remove a lot more than this, this is just a start.

There are various tools to help with this, knip is a good one
2024-08-11 20:43:18 +02:00
a2a5ab488c When exporting a kanban we should export the kanban's main field (#6444)
This PR was created by [GitStart](https://gitstart.com/) to address the
requirements from this ticket:
[TWNTY-6046](https://clients.gitstart.com/twenty/5449/tickets/TWNTY-6046).
This ticket was imported from:
[TWNTY-6046](https://github.com/twentyhq/twenty/issues/6046)

 --- 

### Description

- We are getting the `kanbanFieldMetadataNameState` , get the column
data, and if there is data and the use is on the Kanban view we add the
data to the result

### Refs

#6046

### Demo

<https://jam.dev/c/96f16211-40e4-4b49-a6f5-88f0692fb47a>

Fixes #6046

---------

Co-authored-by: gitstart-twenty <gitstart-twenty@users.noreply.github.com>
Co-authored-by: gitstart-twenty <140154534+gitstart-twenty@users.noreply.github.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
2024-08-09 10:23:06 +02:00
b4e2ada3b0 Fixes Empty Label Identifer Preview in Settings/DataModel/Object/Edit (#6370)
fixes #6143

---------

Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
2024-08-08 18:30:02 +02:00
11a41b3d97 feat: created by email calendar (#6536)
This PR is a followup of #6324 to add support of EMAIL and CALENDAR
source for the created by composite field.
2024-08-07 15:03:06 +02:00
2b311b5f7b Update standard fields (#6532)
In this PR:
- adding Favorites to Tasks and Notes
- fixing inconsistencies between custom object creation and sync of
standard fields of custom objects
- fixing workspaceCacheVersion not used to invalidate existing
datasource
2024-08-04 23:22:41 +02:00
6432ad39b9 feat: add new ACTOR field type and createdBy standard fields (#6324)
This pull request introduces a new `FieldMetadataType` called `ACTOR`.
The primary objective of this new type is to add an extra column to the
following objects: `person`, `company`, `opportunity`, `note`, `task`,
and all custom objects.

This composite type contains three properties:

- `source`
    ```typescript
    export enum FieldActorSource {
      EMAIL = 'EMAIL',
      CALENDAR = 'CALENDAR',
      API = 'API',
      IMPORT = 'IMPORT',
      MANUAL = 'MANUAL',
    }
    ```
- `workspaceMemberId`
- This property can be `undefined` in some cases and refers to the
member who created the record.
- `name`
- Serves as a fallback if the `workspaceMember` is deleted and is used
for other source types like `API`.

### Functionality

The pre-hook system has been updated to allow real-time argument
updates. When a record is created, a pre-hook can now compute and update
the arguments accordingly. This enhancement enables the `createdBy`
field to be populated with the correct values based on the
`authContext`.

The `authContext` now includes:
- An optional User entity
- An optional ApiKey entity
- The workspace entity

This provides access to the necessary data for the `createdBy` field.

In the GraphQL API, only the `source` can be specified in the
`createdBy` input. This allows the front-end to specify the source when
creating records from a CSV file.

### Front-End Handling

On the front-end, `orderBy` and `filter` are only applied to the name
property of the `ACTOR` composite type. Currently, we are unable to
apply these operations to the workspace member relation. This means that
if a workspace member changes their first name or last name, there may
be a mismatch because the name will differ from the new one. The name
displayed on the screen is based on the workspace member entity when
available.

### Missing Components

Currently, this PR does not include a `createdBy` value for the `MAIL`
and `CALENDAR` sources. These records are created in a job, and at
present, we only have access to the workspaceId within the job. To
address this, we should use a function similar to
`loadServiceWithContext`, which was recently removed from `TwentyORM`.
This function would allow us to pass the `authContext` to the jobs
without disrupting existing jobs.
Another PR will be created to handle these cases.

### Related Issues

Fixes issue #5155.

### Additional Notes

This PR doesn't include the migrations of the current records and views.
Everything works properly when the database is reset but this part is
still missing for now. We'll add that in another PR.

- There is a minor issue: front-end tests are broken since this commit:
[80c0fc7ff1).

---------

Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
2024-08-03 15:43:31 +02:00
0dcdda3928 Add updated at column (#6506)
Introduced `updatedAt` column. and fix an existing bug where the field
edition page was crashing because we were trying to compute Date('now')
(param coming from the default value)
2024-08-02 10:03:36 +02:00
80c0fc7ff1 Activity as standard object (#6219)
In this PR I layout the first steps to migrate Activity to a traditional
Standard objects

Since this is a big transition, I'd rather split it into several
deployments / PRs

<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/012e2bbf-9d1b-4723-aaf6-269ef588b050">

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: bosiraphael <71827178+bosiraphael@users.noreply.github.com>
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Faisal-imtiyaz123 <142205282+Faisal-imtiyaz123@users.noreply.github.com>
Co-authored-by: Prateek Jain <prateekj1171998@gmail.com>
2024-07-31 15:36:11 +02:00
8e35edad30 Migrate domainName field from text type to links type (#6410)
Closes #5759.
2024-07-30 11:47:37 +02:00
5a1835e9e0 Improve datasource creation resilience to missing cache 2024-07-27 11:35:47 +02:00
8083175432 Fix object metadata not found (#6416) 2024-07-25 20:34:50 +02:00
4cb83e050f Put workfows behind a feature flag (#6417)
We have recently merged
[#workflow](https://github.com/twentyhq/twenty/pull/6412) but we should
put the workflow standard object behind a feature flag for now

---------

Co-authored-by: bosiraphael <raphael.bosi@gmail.com>
2024-07-25 20:33:31 +02:00
d0201e17ac Fix: query runner orm (#6397)
Fix WorkspaceQueryRunner events using TwentyORM

Fix #6057

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-07-25 13:21:57 +02:00
6c34ef9a14 Simplify ORM (#6373)
This PR refactors the ORM-Manager to simplify and unify the datasource
creation. I'm deprecating all usages if InjectWorkspaceDatasource and
InjectWorkspaceRepository as we can't be sure they are up-to-date
2024-07-23 14:13:16 +02:00
8d33264a7d Migrate fields of deprecated type LINK to type LINKS (#6332)
Closes #5909.

Adding a command to migrate fields of type Link to fields of type Links,
including their data.
2024-07-23 09:57:30 +02:00
d212aedf81 Fix ORM (#6363)
This PR fixes a few bugs on TwentyORM:
- fix many to one relations that were not properly queries
- fix many to one relations that were not properly parsed
- compute datasource (or use from cache) at run-time and do not use
injected one that could be outdated

We still need to refactor it to simplify, I feel the API are too complex
and we have too many cache layers. Also the relation computation part is
very complex and bug prone
2024-07-22 15:07:35 +02:00
d8cadad0fa Deprecate inject workspace repo (#6353) 2024-07-20 00:43:29 +02:00
de20c564c7 feat: add eslint rule for enforcing WorkspaceService naming convention (#6308)
### Description

This PR introduces a custom ESLint rule named
`inject-workspace-repository`. The purpose of this rule is to enforce
naming conventions for files and classes that use the
`@InjectWorkspaceRepository` decorator or include services ending with
`WorkspaceService` in their constructors.

### Rule Overview

The new ESLint rule checks for the following conditions:

1. **File Naming**:
- Only file ending with `.service.ts` or `.workspace-service.ts` are
checked.
- If a file contains a class using the `@InjectWorkspaceRepository`
decorator or a service ending with `WorkspaceService` in the
constructor, the file name must end with `.workspace-service.ts`.

2. **Class Naming**:
- Classes that use the `@InjectWorkspaceRepository` decorator or include
services ending with `WorkspaceService` in their constructors must have
names that end with `WorkspaceService`.

### How It Works

The rule inspects each TypeScript file to ensure that the naming
conventions are adhered to. It specifically looks for:

- Constructor parameters with the `@InjectWorkspaceRepository`
decorator.
- Constructor parameters with a type annotation ending with
`WorkspaceService`.

When such parameters are found, it checks the class name and the file
name to ensure they conform to the expected patterns.

### Example Code

#### Valid Cases

1. **Correct File and Class Name with Decorator**:
    ```typescript
    // Filename: my.workspace-service.ts
    class MyWorkspaceService {
      constructor(@InjectWorkspaceRepository() private repository) {}
    }
    ```

2. **Service Dependency**:
    ```typescript
    // Filename: another.workspace-service.ts
    class AnotherWorkspaceService {
      constructor(private myWorkspaceService: MyWorkspaceService) {}
    }
    ```

#### Invalid Cases

1. **Incorrect Class Name**:
    ```typescript
    // Filename: my.workspace-service.ts
    class MyService {
      constructor(@InjectWorkspaceRepository() private repository) {}
    }
    // Error: Class name should end with 'WorkspaceService'.
    ```

2. **Incorrect File Name**:
    ```typescript
    // Filename: my.service.ts
    class MyWorkspaceService {
      constructor(@InjectWorkspaceRepository() private repository) {}
    }
    // Error: File name should end with '.workspace-service.ts'.
    ```

3. **Incorrect File and Class Name**:
    ```typescript
    // Filename: my.service.ts
    class MyService {
      constructor(@InjectWorkspaceRepository() private repository) {}
    }
    // Error: Class name should end with 'WorkspaceService'.
    // Error: File name should end with '.workspace-service.ts'.
    ```

4. **Incorrect File Type**:
    ```typescript
    // Filename: another.service.ts
    class AnotherService {
      constructor(private myWorkspaceService: MyWorkspaceService) {}
    }
    // Error: Class name should end with 'WorkspaceService'.
    // Error: File name should end with '.workspace-service.ts'.

    ```

5. **Incorrect Class Name with Dependency**:
    ```typescript
    // Filename: another.workspace-service.ts
    class AnotherService {
      constructor(private myWorkspaceService: MyWorkspaceService) {}
    }
    // Error: Class name should end with 'WorkspaceService'.
    ```

### First step

This rule is only a warning for now, and then we'll migrate all the code
that need to be migrated and move from `warn` to `error`.

Fix #6309

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-07-19 19:17:57 +02:00
088d061b3e feat: twenty orm for standard and custom objects (#6178)
### Overview

This PR builds upon #5153, adding the ability to get a repository for
custom objects. The `entitySchema` is now generated for both standard
and custom objects based on metadata stored in the database instead of
the decorated `WorkspaceEntity` in the code. This change ensures that
standard objects with custom fields and relations can also support
custom objects.

### Implementation Details

#### Key Changes:

- **Dynamic Schema Generation:** The `entitySchema` for standard and
custom objects is now dynamically generated from the metadata stored in
the database. This shift allows for greater flexibility and
adaptability, particularly for standard objects with custom fields and
relations.
  
- **Custom Object Repository Retrieval:** A repository for a custom
object can be retrieved using `TwentyORMManager` based on the object's
name. Here's an example of how this can be achieved:

  ```typescript
const repository = await this.twentyORMManager.getRepository('custom');
  /*
* `repository` variable will be typed as follows, ensuring that standard
fields and relations are properly typed:
   * const repository: WorkspaceRepository<CustomWorkspaceEntity & {
   *    [key: string]: any;
   * }>
   */
  const res = await repository.find({});
  ```

Fix #6179

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: Weiko <corentin@twenty.com>
2024-07-19 18:23:52 +02:00
67e2d5c73a Add label identifier to object decorator (#6227)
## Context
LabelIdentifier and ImageIdentifier are metadata info attached to
objectMetadata that are used to display a record in a more readable way.
Those columns point to existing fields that are part of the object.
For example, for a relation picker of a person, we will show a record
using the "name" labelIdentifier and the "avatarUrl" imageIdentifier.
<img width="215" alt="Screenshot 2024-07-11 at 18 45 51"
src="https://github.com/twentyhq/twenty/assets/1834158/488f8294-0d7c-4209-b763-2499716ef29d">

Currently, the FE has a specific logic for company and people objects
and we have a way to update this value via the API for custom objects,
but the code is not flexible enough to change other standard objects.

This PR updates the WorkspaceEntity API so we can now provide the
labelIdentifier and imageIdentifier in the WorkspaceEntity decorator.

Example:
```typescript
@WorkspaceEntity({
  standardId: STANDARD_OBJECT_IDS.activity,
  namePlural: 'activities',
  labelSingular: 'Activity',
  labelPlural: 'Activities',
  description: 'An activity',
  icon: 'IconCheckbox',
  labelIdentifierStandardId: ACTIVITY_STANDARD_FIELD_IDS.title,
})
@WorkspaceIsSystem()
export class ActivityWorkspaceEntity extends BaseWorkspaceEntity {
  @WorkspaceField({
    standardId: ACTIVITY_STANDARD_FIELD_IDS.title,
    type: FieldMetadataType.TEXT,
    label: 'Title',
    description: 'Activity title',
    icon: 'IconNotes',
  })
  title: string;
...
```
2024-07-19 14:24:04 +02:00
11da718482 Refactor connected account module (#6225)
- Refactor connected account module
- Move blocklist into it's own module
- Move contact-creation-manager into it's own module

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-07-12 20:15:33 +02:00
34d13a7b58 Deprecate address standard field (#6087)
Closes #5916

---------

Co-authored-by: Weiko <corentin@twenty.com>
2024-07-10 18:07:18 +02:00
6af1bcd55c [Flexible Schema] Create indexes for join columns (#6165)
## Context
We want to add an index on our foreign keys since PG does not do it for
us. An index can sometimes be expensive and not always meaningful
depending on different usages but in our case we decided to apply an
index for every foreign keys.

```typescript
  @WorkspaceIndex()
  @WorkspaceJoinColumn('author')
  authorId: string;
```
This syntax is valid but since we want to apply it to every join column
I've decided to update the code of WorkspaceJoinColumn so it properly
registers a new index at the same time which is less error-prone.

Note: We had a bug on index name generation since postgres index names
are unique per schema and not table, the object metadata id (hashed) has
been added to the formula that generates the name of the index

## Test
Sync metadata. We have 45 join columns as of today per workspace, we
should see 45 rows inside IndexMetadata table
2024-07-09 14:51:24 +02:00
f458322303 Refactor calendar to use new sync statuses and stages (#6141)
- Refactor calendar modules and some messaging modules to better
organize them by business rules and decouple them
- Work toward a common architecture for the different calendar providers
by introducing interfaces for the drivers
- Modify cron job to use the new sync statuses and stages
2024-07-08 17:01:06 +02:00
5df0ea6466 fix: message queue injection issue (#6126) 2024-07-04 12:22:58 +02:00
95c5602a4e feat: manually implement joinColumn (#6022)
This PR introduce a new decorator named `@WorkspaceJoinColumn`, the goal
of this one is to manually declare the join columns inside the workspace
entities, so we don't have to rely on `ObjectRecord` type.

This decorator can be used that way:

```typescript
  @WorkspaceRelation({
    standardId: ACTIVITY_TARGET_STANDARD_FIELD_IDS.company,
    type: RelationMetadataType.MANY_TO_ONE,
    label: 'Company',
    description: 'ActivityTarget company',
    icon: 'IconBuildingSkyscraper',
    inverseSideTarget: () => CompanyWorkspaceEntity,
    inverseSideFieldKey: 'activityTargets',
  })
  @WorkspaceIsNullable()
  company: Relation<CompanyWorkspaceEntity> | null;

  // The argument is the name of the relation above
  @WorkspaceJoinColumn('company')
  companyId: string | null;
```
2024-06-27 11:41:22 +02:00
7c2e745b45 feat: Dynamic hook registration for WorkspaceQueryHooks (#6008)
#### Overview

This PR introduces a new API for dynamically registering and executing
pre and post query hooks in the Workspace Query Hook system using the
`@WorkspaceQueryHook` decorator. This approach eliminates the need for
manual provider registration, and fix the issue of `undefined` or `null`
repository using `@InjectWorkspaceRepository`.

#### New API

**Define a Hook**

Use the `@WorkspaceQueryHook` decorator to define pre or post hooks:

```typescript
@WorkspaceQueryHook({
  key: `calendarEvent.findMany`,
  scope: Scope.REQUEST,
})
export class CalendarEventFindManyPreQueryHook implements WorkspaceQueryHookInstance {
  async execute(userId: string, workspaceId: string, payload: FindManyResolverArgs): Promise<void> {
    if (!payload?.filter?.id?.eq) {
      throw new BadRequestException('id filter is required');
    }

    // Implement hook logic here
  }
}
```

This API simplifies the registration and execution of query hooks,
providing a more flexible and maintainable approach.

---------

Co-authored-by: Weiko <corentin@twenty.com>
2024-06-25 12:41:46 +02:00
f8c057deea Fix sign up broken because of missing workspace schema (#6013)
Allow workspace datasource factory to return null if the workspace
schema has not been created yet
2024-06-25 10:59:07 +02:00
e13dc7a1fc [FlexibleSchema] Add IndexMetadata decorator (#5981)
## Context
Our Flexible Schema engine dynamically generates entities/tables/APIs
for us but was not flexible enough to build indexes in the DB. With more
and more features involving heavy queries such as Messaging, we are now
adding a new WorkspaceIndex() decorator for our standard objects (will
come later for custom objects). This decorator will give enough
information to the workspace sync metadata manager to generate the
proper migrations that will create or drop indexes on demand.
To be aligned with the rest of the engine, we are adding 2 new tables:
IndexMetadata and IndexFieldMetadata, that will store the info of our
indexes.

## Implementation

```typescript
@WorkspaceEntity({
  standardId: STANDARD_OBJECT_IDS.person,
  namePlural: 'people',
  labelSingular: 'Person',
  labelPlural: 'People',
  description: 'A person',
  icon: 'IconUser',
})
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
  @WorkspaceField({
    standardId: PERSON_STANDARD_FIELD_IDS.email,
    type: FieldMetadataType.EMAIL,
    label: 'Email',
    description: 'Contact’s Email',
    icon: 'IconMail',
  })
  @WorkspaceIndex()
  email: string;
```
By simply adding the WorkspaceIndex decorator, sync-metadata command
will create a new index for that column.
We can also add composite indexes, note that the order is important for
PSQL.
```typescript
@WorkspaceEntity({
  standardId: STANDARD_OBJECT_IDS.person,
  namePlural: 'people',
  labelSingular: 'Person',
  labelPlural: 'People',
  description: 'A person',
  icon: 'IconUser',
})
@WorkspaceIndex(['phone', 'email'])
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```

Currently composite fields and relation fields are not handled by
@WorkspaceIndex() and you will need to use this notation instead
```typescript
@WorkspaceIndex(['companyId', 'nameFirstName'])
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```
<img width="700" alt="Screenshot 2024-06-21 at 15 15 45"
src="https://github.com/twentyhq/twenty/assets/1834158/ac6da1d9-d315-40a4-9ba6-6ab9ae4709d4">

Next step: We might need to implement more complex index expressions,
this is why we have an expression column in IndexMetadata.
What I had in mind for the decorator, still open to discussion
```typescript
@WorkspaceIndex(['nameFirstName', 'nameLastName'], { expression: "$1 || ' ' || $2"})
export class PersonWorkspaceEntity extends BaseWorkspaceEntity {
```

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-06-22 12:39:57 +02:00
0b4bfce324 feat: drop calendar repository (#5824)
This PR is replacing and removing all the raw queries and repositories
with the new `TwentyORM` and injection system using
`@InjectWorkspaceRepository`.
Some logic that was contained inside repositories has been moved to the
services.
In this PR we're only replacing repositories for calendar feature.

---------

Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: bosiraphael <raphael.bosi@gmail.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
2024-06-22 09:26:58 +02:00
d99b9d1d6b feat: Enhancements to MessageQueue Module with Decorators (#5657)
### Overview

This PR introduces significant enhancements to the MessageQueue module
by integrating `@Processor`, `@Process`, and `@InjectMessageQueue`
decorators. These changes streamline the process of defining and
managing queue processors and job handlers, and also allow for
request-scoped handlers, improving compatibility with services that rely
on scoped providers like TwentyORM repositories.

### Key Features

1. **Decorator-based Job Handling**: Use `@Processor` and `@Process`
decorators to define job handlers declaratively.
2. **Request Scope Support**: Job handlers can be scoped per request,
enhancing integration with request-scoped services.

### Usage

#### Defining Processors and Job Handlers

The `@Processor` decorator is used to define a class that processes jobs
for a specific queue. The `@Process` decorator is applied to methods
within this class to define specific job handlers.

##### Example 1: Specific Job Handlers

```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';

@Processor('taskQueue')
export class TaskProcessor {

  @Process('taskA')
  async handleTaskA(job: { id: string, data: any }) {
    console.log(`Handling task A with data:`, job.data);
    // Logic for task A
  }

  @Process('taskB')
  async handleTaskB(job: { id: string, data: any }) {
    console.log(`Handling task B with data:`, job.data);
    // Logic for task B
  }
}
```

In the example above, `TaskProcessor` is responsible for processing jobs
in the `taskQueue`. The `handleTaskA` method will only be called for
jobs with the name `taskA`, while `handleTaskB` will be called for
`taskB` jobs.

##### Example 2: General Job Handler

```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';

@Processor('generalQueue')
export class GeneralProcessor {

  @Process()
  async handleAnyJob(job: { id: string, name: string, data: any }) {
    console.log(`Handling job ${job.name} with data:`, job.data);
    // Logic for any job
  }
}
```

In this example, `GeneralProcessor` handles all jobs in the
`generalQueue`, regardless of the job name. The `handleAnyJob` method
will be invoked for every job added to the `generalQueue`.

#### Adding Jobs to a Queue

You can use the `@InjectMessageQueue` decorator to inject a queue into a
service and add jobs to it.

##### Example:

```typescript
import { Injectable } from '@nestjs/common';
import { InjectMessageQueue, MessageQueue } from 'src/engine/integrations/message-queue';

@Injectable()
export class TaskService {
  constructor(
    @InjectMessageQueue('taskQueue') private readonly taskQueue: MessageQueue,
  ) {}

  async addTaskA(data: any) {
    await this.taskQueue.add('taskA', data);
  }

  async addTaskB(data: any) {
    await this.taskQueue.add('taskB', data);
  }
}
```

In this example, `TaskService` adds jobs to the `taskQueue`. The
`addTaskA` and `addTaskB` methods add jobs named `taskA` and `taskB`,
respectively, to the queue.

#### Using Scoped Job Handlers

To utilize request-scoped job handlers, specify the scope in the
`@Processor` decorator. This is particularly useful for services that
use scoped repositories like those in TwentyORM.

##### Example:

```typescript
import { Processor, Process, InjectMessageQueue, Scope } from 'src/engine/integrations/message-queue';

@Processor({ name: 'scopedQueue', scope: Scope.REQUEST })
export class ScopedTaskProcessor {

  @Process('scopedTask')
  async handleScopedTask(job: { id: string, data: any }) {
    console.log(`Handling scoped task with data:`, job.data);
    // Logic for scoped task, which might use request-scoped services
  }
}
```

Here, the `ScopedTaskProcessor` is associated with `scopedQueue` and
operates with request scope. This setup is essential when the job
handler relies on services that need to be instantiated per request,
such as scoped repositories.

### Migration Notes

- **Decorators**: Refactor job handlers to use `@Processor` and
`@Process` decorators.
- **Request Scope**: Utilize the scope option in `@Processor` if your
job handlers depend on request-scoped services.

Fix #5628

---------

Co-authored-by: Weiko <corentin@twenty.com>
2024-06-17 09:49:37 +02:00
bb6df43d17 fix: twentyORM datasource configuration for ssl (#5529)
We need to specify ssl configuration for TwentyORM datasources when
needed, otherwise connection will be broken.
2024-05-22 11:11:05 +02:00
8b5f79ddbf fix: multiple twenty orm issues & show an example of use (#5439)
This PR is fixing some issues and adding enhancement in TwentyORM:

- [x] Composite fields in nested relations are not formatted properly
- [x] Passing operators like `Any` in `where` condition is breaking the
query
- [x] Ability to auto load workspace-entities based on a regex path

I've also introduced an example of use for `CalendarEventService`:


https://github.com/twentyhq/twenty/pull/5439/files#diff-3a7dffc0dea57345d10e70c648e911f98fe237248bcea124dafa9c8deb1db748R15
2024-05-20 11:01:47 +02:00
f0383e3147 feat: twenty orm sync (#5266)
This PR is updating all object metadata entities with the new
decorators, and deleting the old ones.
This way we can use the new TwentyORM with all the standard objects.

---------

Co-authored-by: Weiko <corentin@twenty.com>
2024-05-15 16:58:47 +02:00
b207d10312 feat: extend twenty orm (#5238)
This PR is a follow up of PR #5153.
This one introduce some changes on how we're querying composite fields.
We can do:

```typescript
export class CompanyService {
  constructor(
    @InjectWorkspaceRepository(CompanyObjectMetadata)
    private readonly companyObjectMetadataRepository: WorkspaceRepository<CompanyObjectMetadata>,
  ) {}

  async companies(): Promise<CompanyObjectMetadata[]> {
    // Old way
    // const companiesFilteredByLinkLabel = await this.companyObjectMetadataRepository.find({
    //   where: { xLinkLabel: 'MyLabel' },
    // });
    // Result will return xLinkLabel property

    // New way
    const companiesFilteredByLinkLabel = await this.companyObjectMetadataRepository.find({
      where: { xLink: { label:  'MyLabel' } },
    });
    // Result will return { xLink: { label: 'MyLabel' } } property instead of  { xLinkLabel: 'MyLabel' }

    return companiesFilteredByLinkLabel;
  }
}
```

Also we can now inject `TwentyORMManage` class to manually create a
repository based on a given `workspaceId` using
`getRepositoryForWorkspace` function that way:

```typescript
export class CompanyService {
  constructor(
    // TwentyORMModule should be initialized
    private readonly twentyORMManager,
  ) {}

  async companies(): Promise<CompanyObjectMetadata[]> {
    const repository = await this.twentyORMManager.getRepositoryForWorkspace(
      '8bb6e872-a71f-4341-82b5-6b56fa81cd77',
      CompanyObjectMetadata,
    );

    const companies = await repository.find();

    return companies;
  }
}
```
2024-05-06 14:12:11 +02:00
e2185448ed Feat/twenty orm (#5153)
## Introduction

This PR introduces "TwentyORM," a custom ORM module designed to
streamline database interactions within our workspace schema, reducing
the need for raw SQL queries. The API mirrors TypeORM's to provide a
familiar interface while integrating enhancements specific to our
project's needs.

To facilitate this integration, new decorators prefixed with `Workspace`
have been implemented. These decorators are used to define entity
metadata more explicitly and are critical in constructing our schema
dynamically.

## New Features

- **Custom ORM System**: Named "TwentyORM," which aligns closely with
TypeORM for ease of use but is tailored to our application's specific
requirements.
- **Decorator-Driven Configuration**: Entities are now configured with
`Workspace`-prefixed decorators that clearly define schema mappings and
relationships directly within the entity classes.
- **Injectable Repositories**: Repositories can be injected similarly to
TypeORM, allowing for flexible and straightforward data management.

## Example Implementations

### Decorated Entity Definitions

Entities are defined with new decorators that outline table and field
metadata, relationships, and constraints. Here are examples of these
implementations:

#### Company Metadata Object

```typescript
@WorkspaceObject({
  standardId: STANDARD_OBJECT_IDS.company,
  namePlural: 'companies',
  labelSingular: 'Company',
  labelPlural: 'Companies',
  description: 'A company',
  icon: 'IconBuildingSkyscraper',
})
export class CompanyObjectMetadata extends BaseObjectMetadata {
  @WorkspaceField({
    standardId: COMPANY_STANDARD_FIELD_IDS.name,
    type: FieldMetadataType.TEXT,
    label: 'Name',
    description: 'The company name',
    icon: 'IconBuildingSkyscraper',
  })
  name: string;

  @WorkspaceField({
    standardId: COMPANY_STANDARD_FIELD_IDS.xLink,
    type: FieldMetadataType.LINK,
    label: 'X',
    description: 'The company Twitter/X account',
    icon: 'IconBrandX',
  })
  @WorkspaceIsNullable()
  xLink: LinkMetadata;

  @WorkspaceField({
    standardId: COMPANY_STANDARD_FIELD_IDS.position,
    type: FieldMetadataType.POSITION,
    label: 'Position',
    description: 'Company record position',
    icon: 'IconHierarchy2',
  })
  @WorkspaceIsSystem()
  @WorkspaceIsNullable()
  position: number;

  @WorkspaceRelation({
    standardId: COMPANY_STANDARD_FIELD_IDS.accountOwner,
    label: 'Account Owner',
    description: 'Your team member responsible for managing the company account',
    type: RelationMetadataType.MANY_TO_ONE,
    inverseSideTarget: () => WorkspaceMemberObjectMetadata,
    inverseSideFieldKey: 'accountOwnerForCompanies',
    onDelete: RelationOnDeleteAction.SET_NULL,
  })
  @WorkspaceIsNullable()
  accountOwner: WorkspaceMemberObjectMetadata;
}
```

#### Workspace Member Metadata Object

```typescript
@WorkspaceObject({
  standardId: STANDARD_OBJECT_IDS.workspaceMember,
  namePlural: 'workspaceMembers',
  labelSingular: 'Workspace Member',
  labelPlural: 'Workspace Members',
  description: 'A workspace member',
  icon: 'IconUserCircle',
})
@WorkspaceIsSystem()
@WorkspaceIsNotAuditLogged()
export class WorkspaceMemberObjectMetadata extends BaseObjectMetadata {
  @WorkspaceField({
    standardId: WORKSPACE_MEMBER_STANDARD_FIELD_IDS.name,
    type: FieldMetadataType.FULL_NAME,
    label: 'Name',
    description: 'Workspace member name',
    icon: 'IconCircleUser',
  })
  name: FullNameMetadata;

  @WorkspaceRelation({
    standardId: WORKSPACE_MEMBER_STANDARD_FIELD_IDS.accountOwnerForCompanies,
    label: 'Account Owner For Companies',
    description: 'Account owner for companies',
    icon: 'IconBriefcase',
    type: RelationMetadataType.ONE_TO_MANY,
    inverseSideTarget: () => CompanyObjectMetadata,
    inverseSideFieldKey: 'accountOwner',
    onDelete: RelationOnDeleteAction.SET_NULL,
  })
  accountOwnerForCompanies: Relation

<CompanyObjectMetadata[]>;
}
```

### Injectable Repository Usage

Repositories can be directly injected into services, allowing for
streamlined query operations:

```typescript
export class CompanyService {
  constructor(
    @InjectWorkspaceRepository(CompanyObjectMetadata)
    private readonly companyObjectMetadataRepository: WorkspaceRepository<CompanyObjectMetadata>,
  ) {}

  async companies(): Promise<CompanyObjectMetadata[]> {
    // Example queries demonstrating simple and relation-loaded operations
    const simpleCompanies = await this.companyObjectMetadataRepository.find({});
    const companiesWithOwners = await this.companyObjectMetadataRepository.find({
      relations: ['accountOwner'],
    });
    const companiesFilteredByLinkLabel = await this.companyObjectMetadataRepository.find({
      where: { xLinkLabel: 'MyLabel' },
    });

    return companiesFilteredByLinkLabel;
  }
}
```

## Conclusions

This PR sets the foundation for a decorator-driven ORM layer that
simplifies data interactions and supports complex entity relationships
while maintaining clean and manageable code architecture. This is not
finished yet, and should be extended.
All the standard objects needs to be migrated and all the module using
the old decorators too.

---------

Co-authored-by: Weiko <corentin@twenty.com>
2024-04-29 16:47:42 +02:00