This PR is replacing and removing all the raw queries and repositories
with the new `TwentyORM` and injection system using
`@InjectWorkspaceRepository`.
Some logic that was contained inside repositories has been moved to the
services.
In this PR we're only replacing repositories for calendar feature.
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: bosiraphael <raphael.bosi@gmail.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
- Remove filters from metadata rest api
- add limite before and after parameters for metadata
- remove update from metadata relations
- fix typing issue
- fix naming
- fix before parameter
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Filtering relations is not allowed
(see`packages/twenty-server/src/engine/metadata-modules/relation-metadata/dtos/relation-metadata.dto.ts`)
so we remove filtering for find many relation
we also fixed some bug in result structure and metadata open-api schema
### Overview
This PR introduces significant enhancements to the MessageQueue module
by integrating `@Processor`, `@Process`, and `@InjectMessageQueue`
decorators. These changes streamline the process of defining and
managing queue processors and job handlers, and also allow for
request-scoped handlers, improving compatibility with services that rely
on scoped providers like TwentyORM repositories.
### Key Features
1. **Decorator-based Job Handling**: Use `@Processor` and `@Process`
decorators to define job handlers declaratively.
2. **Request Scope Support**: Job handlers can be scoped per request,
enhancing integration with request-scoped services.
### Usage
#### Defining Processors and Job Handlers
The `@Processor` decorator is used to define a class that processes jobs
for a specific queue. The `@Process` decorator is applied to methods
within this class to define specific job handlers.
##### Example 1: Specific Job Handlers
```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';
@Processor('taskQueue')
export class TaskProcessor {
@Process('taskA')
async handleTaskA(job: { id: string, data: any }) {
console.log(`Handling task A with data:`, job.data);
// Logic for task A
}
@Process('taskB')
async handleTaskB(job: { id: string, data: any }) {
console.log(`Handling task B with data:`, job.data);
// Logic for task B
}
}
```
In the example above, `TaskProcessor` is responsible for processing jobs
in the `taskQueue`. The `handleTaskA` method will only be called for
jobs with the name `taskA`, while `handleTaskB` will be called for
`taskB` jobs.
##### Example 2: General Job Handler
```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';
@Processor('generalQueue')
export class GeneralProcessor {
@Process()
async handleAnyJob(job: { id: string, name: string, data: any }) {
console.log(`Handling job ${job.name} with data:`, job.data);
// Logic for any job
}
}
```
In this example, `GeneralProcessor` handles all jobs in the
`generalQueue`, regardless of the job name. The `handleAnyJob` method
will be invoked for every job added to the `generalQueue`.
#### Adding Jobs to a Queue
You can use the `@InjectMessageQueue` decorator to inject a queue into a
service and add jobs to it.
##### Example:
```typescript
import { Injectable } from '@nestjs/common';
import { InjectMessageQueue, MessageQueue } from 'src/engine/integrations/message-queue';
@Injectable()
export class TaskService {
constructor(
@InjectMessageQueue('taskQueue') private readonly taskQueue: MessageQueue,
) {}
async addTaskA(data: any) {
await this.taskQueue.add('taskA', data);
}
async addTaskB(data: any) {
await this.taskQueue.add('taskB', data);
}
}
```
In this example, `TaskService` adds jobs to the `taskQueue`. The
`addTaskA` and `addTaskB` methods add jobs named `taskA` and `taskB`,
respectively, to the queue.
#### Using Scoped Job Handlers
To utilize request-scoped job handlers, specify the scope in the
`@Processor` decorator. This is particularly useful for services that
use scoped repositories like those in TwentyORM.
##### Example:
```typescript
import { Processor, Process, InjectMessageQueue, Scope } from 'src/engine/integrations/message-queue';
@Processor({ name: 'scopedQueue', scope: Scope.REQUEST })
export class ScopedTaskProcessor {
@Process('scopedTask')
async handleScopedTask(job: { id: string, data: any }) {
console.log(`Handling scoped task with data:`, job.data);
// Logic for scoped task, which might use request-scoped services
}
}
```
Here, the `ScopedTaskProcessor` is associated with `scopedQueue` and
operates with request scope. This setup is essential when the job
handler relies on services that need to be instantiated per request,
such as scoped repositories.
### Migration Notes
- **Decorators**: Refactor job handlers to use `@Processor` and
`@Process` decorators.
- **Request Scope**: Utilize the scope option in `@Processor` if your
job handlers depend on request-scoped services.
Fix#5628
---------
Co-authored-by: Weiko <corentin@twenty.com>
- Improve the rest api by introducing startingAfter/endingBefore (we
previously had lastCursor), and moving pageInfo/totalCount outside of
the data object.
- Fix broken GraphQL playground on website
- Improve analytics by sending server url
* Remove relations where they cannot be used
* Removed duplicated schema for findMany
* Reuse schema for Relation variant to reduce size of sent json object
closes#5778
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: martmull <martmull@hotmail.fr>
In this PR, I'm mainly doing two things:
- uniformizing messaging-messages-import and
messaging-message-list-fetch behaviors (cron.job and job)
- improving performances of these cron.jobs by not triggering the jobs
if the stage is not relevant
- making sure these jobs have same signature (workspaceId +
messageChannelId)
First step for creating credentials for database proxy.
In next PRs:
- When calling endpoint, create database Postgres on proxy server
- Setup user on database using postgresCredentials
- Build remote server on DB to access workspace data
In this PR, I'm refactoring the messaging module into smaller pieces
that have **ONE** responsibility: import messages, clean messages,
handle message participant creation, instead of having ~30 modules (1
per service, jobs, cron, ...). This is mandatory to start introducing
drivers (gmails, office365, ...) IMO. It is too difficult to enforce
common interfaces as we have too many interfaces (30 modules...). All
modules should not be exposed
Right now, we have services that are almost functions:
do-that-and-this.service.ts / do-that-and-this.module.ts
I believe we should have something more organized at a high level and it
does not matter that much if we have a bit of code duplicates.
Note that the proposal is not fully implemented in the current PR that
has only focused on messaging folder (biggest part)
Here is the high level proposal:
- connected-account: token-refresher
- blocklist
- messaging: message-importer, message-cleaner, message-participants,
... (right now I'm keeping a big messaging-common but this will
disappear see below)
- calendar: calendar-importer, calendar-cleaner, ...
Consequences:
1) It's OK to re-implement several times some things. Example:
- error handling in connected-account, messaging, and calendar instead
of trying to unify. They are actually different error handling. The only
things that might be in common is the GmailError => CommonError parsing
and I'm not even sure it makes a lot of sense as these 3 apis might have
different format actually
- auto-creation. Calendar and Messaging could actually have different
rules
2) **We should not have circular dependencies:**
- I believe this was the reason why we had so many modules, to be able
to cherry pick the one we wanted to avoid circular deps. This is not the
right approach IMO, we need architect the whole messaging by defining
high level blocks that won't have circular dependencies by design. If we
encounter one, we should rethink and break the block in a way that makes
sense.
- ex: connected-account.resolver is not in the same module as
token-refresher. ==> connected-account.resolver => message-importer (as
we trigger full sync job when we connect an account) => token-refresher
(as we refresh token on message import).
connected-account.resolver and token-refresher both in connected-account
folder but should be in different modules. Otherwise it's a circular
dependency. It does not mean that we should create 1 module per service
as it was done before
In a nutshell: The code needs to be thought in term of reponsibilities
and in a way that enforce high level interfaces (and avoid circular
dependencies)
Bonus: As you can see, this code is also removing a lot of code because
of the removal of many .module.ts (also because I'm removing the sync
scripts v2 feature flag end removing old code)
Bonus: I have prefixed services name with Messaging to improve dev xp.
GmailErrorHandler could be different between MessagingGmailErrorHandler
and CalendarGmailErrorHandler for instance
Fix a bug introduced in [this
PR](https://github.com/twentyhq/twenty/pull/5254/files)
When a subscription is created, we need to create the subscription,
#5254 return if no subscription is created so the sub can never be
created at all
This PR fixes that
This PR is updating all object metadata entities with the new
decorators, and deleting the old ones.
This way we can use the new TwentyORM with all the standard objects.
---------
Co-authored-by: Weiko <corentin@twenty.com>
When user is deleting its account on a specific workspace, we remove it
as if it was a workspaceMember, and if no workspaceMember remains, we
delete the workspace and the associated stripe subscription
# This PR
- Fix#5021
- Migrates `passwordResetToken` and `passwordResetTokenExpiresAt` fields
from `core.users` to `core.appToken`
- Marks those fields as `deprecated` so we can remove them later if we
are happy with the transition -- I took this decision on my own,
@FellipeMTX let me know what you think about it, we can also remove them
straight away if you think it's better
- Fixed the `database:migration` script from the `twenty-server` to:
```json
"database:migrate": {
"executor": "nx:run-commands",
"dependsOn": ["build"], // added this line
"options": {
"cwd": "packages/twenty-server",
"commands": [
"nx typeorm -- migration:run -d src/database/typeorm/metadata/metadata.datasource",
"nx typeorm -- migration:run -d src/database/typeorm/core/core.datasource"
],
"parallel": false
}
},
```
The migration script wasn't running because the builds were not executed
- [x] Added unit tests for the token.service file's changes
Looking forward to hearing feedback from you
cc: @charlesBochet
---------
Co-authored-by: Weiko <corentin@twenty.com>
Adding stripe integration by making the server logic independent of the
input fields:
- query factories (remote server, foreign data wrapper, foreign table)
to loop on fields and values without hardcoding the names of the fields
- adding stripe input and type
- add the logic to handle static schema. Simply creating a big object to
store into the server
Additional work:
- rename username field to user. This is the input intended for postgres
user mapping and we now need a matching by name
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
## Introduction
This PR introduces "TwentyORM," a custom ORM module designed to
streamline database interactions within our workspace schema, reducing
the need for raw SQL queries. The API mirrors TypeORM's to provide a
familiar interface while integrating enhancements specific to our
project's needs.
To facilitate this integration, new decorators prefixed with `Workspace`
have been implemented. These decorators are used to define entity
metadata more explicitly and are critical in constructing our schema
dynamically.
## New Features
- **Custom ORM System**: Named "TwentyORM," which aligns closely with
TypeORM for ease of use but is tailored to our application's specific
requirements.
- **Decorator-Driven Configuration**: Entities are now configured with
`Workspace`-prefixed decorators that clearly define schema mappings and
relationships directly within the entity classes.
- **Injectable Repositories**: Repositories can be injected similarly to
TypeORM, allowing for flexible and straightforward data management.
## Example Implementations
### Decorated Entity Definitions
Entities are defined with new decorators that outline table and field
metadata, relationships, and constraints. Here are examples of these
implementations:
#### Company Metadata Object
```typescript
@WorkspaceObject({
standardId: STANDARD_OBJECT_IDS.company,
namePlural: 'companies',
labelSingular: 'Company',
labelPlural: 'Companies',
description: 'A company',
icon: 'IconBuildingSkyscraper',
})
export class CompanyObjectMetadata extends BaseObjectMetadata {
@WorkspaceField({
standardId: COMPANY_STANDARD_FIELD_IDS.name,
type: FieldMetadataType.TEXT,
label: 'Name',
description: 'The company name',
icon: 'IconBuildingSkyscraper',
})
name: string;
@WorkspaceField({
standardId: COMPANY_STANDARD_FIELD_IDS.xLink,
type: FieldMetadataType.LINK,
label: 'X',
description: 'The company Twitter/X account',
icon: 'IconBrandX',
})
@WorkspaceIsNullable()
xLink: LinkMetadata;
@WorkspaceField({
standardId: COMPANY_STANDARD_FIELD_IDS.position,
type: FieldMetadataType.POSITION,
label: 'Position',
description: 'Company record position',
icon: 'IconHierarchy2',
})
@WorkspaceIsSystem()
@WorkspaceIsNullable()
position: number;
@WorkspaceRelation({
standardId: COMPANY_STANDARD_FIELD_IDS.accountOwner,
label: 'Account Owner',
description: 'Your team member responsible for managing the company account',
type: RelationMetadataType.MANY_TO_ONE,
inverseSideTarget: () => WorkspaceMemberObjectMetadata,
inverseSideFieldKey: 'accountOwnerForCompanies',
onDelete: RelationOnDeleteAction.SET_NULL,
})
@WorkspaceIsNullable()
accountOwner: WorkspaceMemberObjectMetadata;
}
```
#### Workspace Member Metadata Object
```typescript
@WorkspaceObject({
standardId: STANDARD_OBJECT_IDS.workspaceMember,
namePlural: 'workspaceMembers',
labelSingular: 'Workspace Member',
labelPlural: 'Workspace Members',
description: 'A workspace member',
icon: 'IconUserCircle',
})
@WorkspaceIsSystem()
@WorkspaceIsNotAuditLogged()
export class WorkspaceMemberObjectMetadata extends BaseObjectMetadata {
@WorkspaceField({
standardId: WORKSPACE_MEMBER_STANDARD_FIELD_IDS.name,
type: FieldMetadataType.FULL_NAME,
label: 'Name',
description: 'Workspace member name',
icon: 'IconCircleUser',
})
name: FullNameMetadata;
@WorkspaceRelation({
standardId: WORKSPACE_MEMBER_STANDARD_FIELD_IDS.accountOwnerForCompanies,
label: 'Account Owner For Companies',
description: 'Account owner for companies',
icon: 'IconBriefcase',
type: RelationMetadataType.ONE_TO_MANY,
inverseSideTarget: () => CompanyObjectMetadata,
inverseSideFieldKey: 'accountOwner',
onDelete: RelationOnDeleteAction.SET_NULL,
})
accountOwnerForCompanies: Relation
<CompanyObjectMetadata[]>;
}
```
### Injectable Repository Usage
Repositories can be directly injected into services, allowing for
streamlined query operations:
```typescript
export class CompanyService {
constructor(
@InjectWorkspaceRepository(CompanyObjectMetadata)
private readonly companyObjectMetadataRepository: WorkspaceRepository<CompanyObjectMetadata>,
) {}
async companies(): Promise<CompanyObjectMetadata[]> {
// Example queries demonstrating simple and relation-loaded operations
const simpleCompanies = await this.companyObjectMetadataRepository.find({});
const companiesWithOwners = await this.companyObjectMetadataRepository.find({
relations: ['accountOwner'],
});
const companiesFilteredByLinkLabel = await this.companyObjectMetadataRepository.find({
where: { xLinkLabel: 'MyLabel' },
});
return companiesFilteredByLinkLabel;
}
}
```
## Conclusions
This PR sets the foundation for a decorator-driven ORM layer that
simplifies data interactions and supports complex entity relationships
while maintaining clean and manageable code architecture. This is not
finished yet, and should be extended.
All the standard objects needs to be migrated and all the module using
the old decorators too.
---------
Co-authored-by: Weiko <corentin@twenty.com>
New strategy:
- add settings field on FieldMetadata. Contains a boolean isIdField and
for numbers, a precision
- if idField, the graphql scalar returned will be a GraphQL id. This
will allow the app to work even for ids that are not uuid
- remove globals dateScalar and numberScalar modes. These were not used
- set limit as Integer
- check manually in query runner mutations that we send a valid id
Todo left:
- remove WorkspaceBuildSchemaOptions since this is not used anymore.
Will do in another PR
---------
Co-authored-by: Thomas Trompette <thomast@twenty.com>
Co-authored-by: Weiko <corentin@twenty.com>
## Description
This PR adds recaptcha on login form. One can add any one of three
recaptcha vendor -
1. Google Recaptcha -
https://developers.google.com/recaptcha/docs/v3#programmatically_invoke_the_challenge
2. HCaptcha -
https://docs.hcaptcha.com/invisible#programmatically-invoke-the-challenge
3. Turnstile -
https://developers.cloudflare.com/turnstile/get-started/client-side-rendering/#execution-modes
### Issue
- #3546
### Environment variables -
1. `CAPTCHA_DRIVER` - `google-recaptcha` | `hcaptcha` | `turnstile`
2. `CAPTCHA_SITE_KEY` - site key
3. `CAPTCHA_SECRET_KEY` - secret key
### Engineering choices
1. If some of the above env variable provided, then, backend generates
an error -
<img width="990" alt="image"
src="https://github.com/twentyhq/twenty/assets/60139930/9fb00fab-9261-4ff3-b23e-2c2e06f1bf89">
Please note that login/signup form will keep working as expected.
2. I'm using a Captcha guard that intercepts the request. If
"captchaToken" is present in the body and all env is set, then, the
captcha token is verified by backend through the service.
3. One can use this guard on any resolver to protect it by the captcha.
4. On frontend, two hooks `useGenerateCaptchaToken` and
`useInsertCaptchaScript` is created. `useInsertCaptchaScript` adds the
respective captcha JS script on frontend. `useGenerateCaptchaToken`
returns a function that one can use to trigger captcha token generation
programatically. This allows one to generate token keeping recaptcha
invisible.
### Note
This PR contains some changes in unrelated files like indentation,
spacing, inverted comma etc. I ran "yarn nx fmt:fix twenty-front" and
"yarn nx lint twenty-front -- --fix".
### Screenshots
<img width="869" alt="image"
src="https://github.com/twentyhq/twenty/assets/60139930/a75f5677-9b66-47f7-9730-4ec916073f8c">
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
Previously we had to create a separate API key to give access to chrome
extension so we can make calls to the DB. This PR includes logic to
initiate a oauth flow with PKCE method which redirects to the
`Authorise` screen to give access to server tokens.
Implemented in this PR-
1. make `redirectUrl` a non-nullable parameter
2. Add `NODE_ENV` to environment variable service
3. new env variable `CHROME_EXTENSION_REDIRECT_URL` on server side
4. strict checks for redirectUrl
5. try catch blocks on utils db query methods
6. refactor Apollo Client to handle `unauthorized` condition
7. input field to enter server url (for self-hosting)
8. state to show user if its already connected
9. show error if oauth flow is cancelled by user
Follow up PR -
Renew token logic
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
Refactored the code to introduce two different concepts:
- AuditLogs (immutable, raw data)
- TimelineActivities (user-friendly, transformed data)
Still some work needed:
- Add message, files, calendar events to timeline (~2 hours if done
naively)
- Refactor repository to try to abstract concept when we can (tbd, wait
for Twenty ORM)
- Introduce ability to display child timelines on parent timeline with
filtering (~2 days)
- Improve UI: add links to open note/task, improve diff display, etc
(half a day)
- Decide the path forward for Task vs Notes: either introduce a new
field type "Record Type" and start going into that direction ; or split
in two objects?
- Trigger updates when a field is changed (will be solved by real-time /
websockets: 2 weeks)
- Integrate behavioral events (1 day for POC, 1 week for
clean/documented)
<img width="1248" alt="Screenshot 2024-04-12 at 09 24 49"
src="https://github.com/twentyhq/twenty/assets/6399865/9428db1a-ab2b-492c-8b0b-d4d9a36e81fa">
