admin/twenty
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

feat: Enhancements to MessageQueue Module with Decorators (#5657)

Browse Source 
### Overview

This PR introduces significant enhancements to the MessageQueue module
by integrating `@Processor`, `@Process`, and `@InjectMessageQueue`
decorators. These changes streamline the process of defining and
managing queue processors and job handlers, and also allow for
request-scoped handlers, improving compatibility with services that rely
on scoped providers like TwentyORM repositories.

### Key Features

1. **Decorator-based Job Handling**: Use `@Processor` and `@Process`
decorators to define job handlers declaratively.
2. **Request Scope Support**: Job handlers can be scoped per request,
enhancing integration with request-scoped services.

### Usage

#### Defining Processors and Job Handlers

The `@Processor` decorator is used to define a class that processes jobs
for a specific queue. The `@Process` decorator is applied to methods
within this class to define specific job handlers.

##### Example 1: Specific Job Handlers

```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';

@Processor('taskQueue')
export class TaskProcessor {

  @Process('taskA')
  async handleTaskA(job: { id: string, data: any }) {
    console.log(`Handling task A with data:`, job.data);
    // Logic for task A
  }

  @Process('taskB')
  async handleTaskB(job: { id: string, data: any }) {
    console.log(`Handling task B with data:`, job.data);
    // Logic for task B
  }
}
```

In the example above, `TaskProcessor` is responsible for processing jobs
in the `taskQueue`. The `handleTaskA` method will only be called for
jobs with the name `taskA`, while `handleTaskB` will be called for
`taskB` jobs.

##### Example 2: General Job Handler

```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';

@Processor('generalQueue')
export class GeneralProcessor {

  @Process()
  async handleAnyJob(job: { id: string, name: string, data: any }) {
    console.log(`Handling job ${job.name} with data:`, job.data);
    // Logic for any job
  }
}
```

In this example, `GeneralProcessor` handles all jobs in the
`generalQueue`, regardless of the job name. The `handleAnyJob` method
will be invoked for every job added to the `generalQueue`.

#### Adding Jobs to a Queue

You can use the `@InjectMessageQueue` decorator to inject a queue into a
service and add jobs to it.

##### Example:

```typescript
import { Injectable } from '@nestjs/common';
import { InjectMessageQueue, MessageQueue } from 'src/engine/integrations/message-queue';

@Injectable()
export class TaskService {
  constructor(
    @InjectMessageQueue('taskQueue') private readonly taskQueue: MessageQueue,
  ) {}

  async addTaskA(data: any) {
    await this.taskQueue.add('taskA', data);
  }

  async addTaskB(data: any) {
    await this.taskQueue.add('taskB', data);
  }
}
```

In this example, `TaskService` adds jobs to the `taskQueue`. The
`addTaskA` and `addTaskB` methods add jobs named `taskA` and `taskB`,
respectively, to the queue.

#### Using Scoped Job Handlers

To utilize request-scoped job handlers, specify the scope in the
`@Processor` decorator. This is particularly useful for services that
use scoped repositories like those in TwentyORM.

##### Example:

```typescript
import { Processor, Process, InjectMessageQueue, Scope } from 'src/engine/integrations/message-queue';

@Processor({ name: 'scopedQueue', scope: Scope.REQUEST })
export class ScopedTaskProcessor {

  @Process('scopedTask')
  async handleScopedTask(job: { id: string, data: any }) {
    console.log(`Handling scoped task with data:`, job.data);
    // Logic for scoped task, which might use request-scoped services
  }
}
```

Here, the `ScopedTaskProcessor` is associated with `scopedQueue` and
operates with request scope. This setup is essential when the job
handler relies on services that need to be instantiated per request,
such as scoped repositories.

### Migration Notes

- **Decorators**: Refactor job handlers to use `@Processor` and
`@Process` decorators.
- **Request Scope**: Utilize the scope option in `@Processor` if your
job handlers depend on request-scoped services.

Fix #5628

---------

Co-authored-by: Weiko <corentin@twenty.com>
This commit is contained in:
Jérémy M
2024-06-17 09:49:37 +02:00
committed by GitHub
parent 605945bd42
commit d99b9d1d6b
92 changed files with 955 additions and 524 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
packages/twenty-server/src/engine/api/graphql/workspace-query-runner/commands/0-20-record-position-backfill.command.ts
View File

@ -1,11 +1,10 @@
import { Inject } from '@nestjs/common';
import { Command, CommandRunner, Option } from 'nest-commander';
import {
RecordPositionBackfillJob,
RecordPositionBackfillJobData,
} from 'src/engine/api/graphql/workspace-query-runner/jobs/record-position-backfill.job';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
@ -20,7 +19,7 @@ export type RecordPositionBackfillCommandOptions = {
})
export class RecordPositionBackfillCommand extends CommandRunner {
constructor(
@Inject(MessageQueue.recordPositionBackfillQueue)
@InjectMessageQueue(MessageQueue.recordPositionBackfillQueue)
private readonly messageQueueService: MessageQueueService,
) {
super();

15
packages/twenty-server/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.ts
View File

@ -1,6 +1,5 @@
import { Inject, Injectable, Logger } from '@nestjs/common';
import { Logger } from '@nestjs/common';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { ObjectMetadataInterface } from 'src/engine/metadata-modules/field-metadata/interfaces/object-metadata.interface';
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
@ -11,6 +10,9 @@ import {
CallWebhookJob,
CallWebhookJobData,
} from 'src/engine/api/graphql/workspace-query-runner/jobs/call-webhook.job';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
import { Processor } from 'src/engine/integrations/message-queue/decorators/processor.decorator';
import { Process } from 'src/engine/integrations/message-queue/decorators/process.decorator';
export enum CallWebhookJobsJobOperation {
create = 'create',
@ -25,19 +27,18 @@ export type CallWebhookJobsJobData = {
operation: CallWebhookJobsJobOperation;
};
@Injectable()
export class CallWebhookJobsJob
implements MessageQueueJob<CallWebhookJobsJobData>
{
@Processor(MessageQueue.webhookQueue)
export class CallWebhookJobsJob {
private readonly logger = new Logger(CallWebhookJobsJob.name);
constructor(
private readonly workspaceDataSourceService: WorkspaceDataSourceService,
private readonly dataSourceService: DataSourceService,
@Inject(MessageQueue.webhookQueue)
@InjectMessageQueue(MessageQueue.webhookQueue)
private readonly messageQueueService: MessageQueueService,
) {}
@Process(CallWebhookJobsJob.name)
async handle(data: CallWebhookJobsJobData): Promise<void> {
const dataSourceMetadata =
await this.dataSourceService.getLastDataSourceMetadataFromWorkspaceIdOrFail(

11
packages/twenty-server/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook.job.ts
View File

@ -1,7 +1,9 @@
import { Injectable, Logger } from '@nestjs/common';
import { Logger } from '@nestjs/common';
import { HttpService } from '@nestjs/axios';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { Processor } from 'src/engine/integrations/message-queue/decorators/processor.decorator';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { Process } from 'src/engine/integrations/message-queue/decorators/process.decorator';
export type CallWebhookJobData = {
targetUrl: string;
@ -13,12 +15,13 @@ export type CallWebhookJobData = {
record: any;
};
@Injectable()
export class CallWebhookJob implements MessageQueueJob<CallWebhookJobData> {
@Processor(MessageQueue.webhookQueue)
export class CallWebhookJob {
private readonly logger = new Logger(CallWebhookJob.name);
constructor(private readonly httpService: HttpService) {}
@Process(CallWebhookJob.name)
async handle(data: CallWebhookJobData): Promise<void> {
try {
await this.httpService.axiosRef.post(data.targetUrl, data);

14
packages/twenty-server/src/engine/api/graphql/workspace-query-runner/jobs/record-position-backfill.job.ts
View File

@ -1,22 +1,20 @@
import { Injectable } from '@nestjs/common';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { RecordPositionBackfillService } from 'src/engine/api/graphql/workspace-query-runner/services/record-position-backfill-service';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { Processor } from 'src/engine/integrations/message-queue/decorators/processor.decorator';
import { Process } from 'src/engine/integrations/message-queue/decorators/process.decorator';
export type RecordPositionBackfillJobData = {
workspaceId: string;
dryRun: boolean;
};
@Injectable()
export class RecordPositionBackfillJob
implements MessageQueueJob<RecordPositionBackfillJobData>
{
@Processor(MessageQueue.recordPositionBackfillQueue)
export class RecordPositionBackfillJob {
constructor(
private readonly recordPositionBackfillService: RecordPositionBackfillService,
) {}
@Process(RecordPositionBackfillJob.name)
async handle(data: RecordPositionBackfillJobData): Promise<void> {
this.recordPositionBackfillService.backfill(data.workspaceId, data.dryRun);
}

15
packages/twenty-server/src/engine/api/graphql/workspace-query-runner/jobs/workspace-query-runner-job.module.ts
View File

@ -15,19 +15,6 @@ import { WorkspaceDataSourceModule } from 'src/engine/workspace-datasource/works
RecordPositionBackfillModule,
HttpModule,
],
providers: [
{
provide: CallWebhookJobsJob.name,
useClass: CallWebhookJobsJob,
},
{
provide: CallWebhookJob.name,
useClass: CallWebhookJob,
},
{
provide: RecordPositionBackfillJob.name,
useClass: RecordPositionBackfillJob,
},
],
providers: [CallWebhookJobsJob, CallWebhookJob, RecordPositionBackfillJob],
})
export class WorkspaceQueryRunnerJobModule {}

5
packages/twenty-server/src/engine/api/graphql/workspace-query-runner/listeners/entity-events-to-db.listener.ts
View File

@ -1,4 +1,4 @@
import { Inject, Injectable } from '@nestjs/common';
import { Injectable } from '@nestjs/common';
import { OnEvent } from '@nestjs/event-emitter';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
@ -8,12 +8,13 @@ import { objectRecordChangedValues } from 'src/engine/integrations/event-emitter
import { ObjectRecordUpdateEvent } from 'src/engine/integrations/event-emitter/types/object-record-update.event';
import { ObjectRecordBaseEvent } from 'src/engine/integrations/event-emitter/types/object-record.base.event';
import { UpsertTimelineActivityFromInternalEvent } from 'src/modules/timeline/jobs/upsert-timeline-activity-from-internal-event.job';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
import { CreateAuditLogFromInternalEvent } from 'src/modules/timeline/jobs/create-audit-log-from-internal-event';
@Injectable()
export class EntityEventsToDbListener {
constructor(
@Inject(MessageQueue.entityEventsToDbQueue)
@InjectMessageQueue(MessageQueue.entityEventsToDbQueue)
private readonly messageQueueService: MessageQueueService,
) {}

4
packages/twenty-server/src/engine/api/graphql/workspace-query-runner/workspace-query-runner.service.ts
View File

@ -1,6 +1,5 @@
import {
BadRequestException,
Inject,
Injectable,
Logger,
RequestTimeoutException,
@ -52,6 +51,7 @@ import { assertMutationNotOnRemoteObject } from 'src/engine/metadata-modules/obj
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
import { assertIsValidUuid } from 'src/engine/api/graphql/workspace-query-runner/utils/assert-is-valid-uuid.util';
import { isQueryTimeoutError } from 'src/engine/utils/query-timeout.util';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
import { WorkspaceQueryRunnerOptions } from './interfaces/query-runner-option.interface';
import {
@ -72,7 +72,7 @@ export class WorkspaceQueryRunnerService {
private readonly workspaceDataSourceService: WorkspaceDataSourceService,
private readonly queryRunnerArgsFactory: QueryRunnerArgsFactory,
private readonly queryResultGettersFactory: QueryResultGettersFactory,
@Inject(MessageQueue.webhookQueue)
@InjectMessageQueue(MessageQueue.webhookQueue)
private readonly messageQueueService: MessageQueueService,
private readonly eventEmitter: EventEmitter2,
private readonly workspacePreQueryHookService: WorkspacePreQueryHookService,

7
packages/twenty-server/src/engine/core-modules/auth/services/google-apis.service.ts
View File

@ -1,4 +1,4 @@
import { Injectable, Inject } from '@nestjs/common';
import { Injectable } from '@nestjs/common';
import { EntityManager } from 'typeorm';
import { v4 } from 'uuid';
@ -34,15 +34,16 @@ import {
MessagingMessageListFetchJob,
MessagingMessageListFetchJobData,
} from 'src/modules/messaging/message-import-manager/jobs/messaging-message-list-fetch.job';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
@Injectable()
export class GoogleAPIsService {
constructor(
private readonly dataSourceService: DataSourceService,
private readonly typeORMService: TypeORMService,
@Inject(MessageQueue.messagingQueue)
@InjectMessageQueue(MessageQueue.messagingQueue)
private readonly messageQueueService: MessageQueueService,
@Inject(MessageQueue.calendarQueue)
@InjectMessageQueue(MessageQueue.calendarQueue)
private readonly calendarQueueService: MessageQueueService,
private readonly environmentService: EnvironmentService,
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)

16
packages/twenty-server/src/engine/core-modules/billing/jobs/update-subscription.job.ts
View File

@ -1,22 +1,24 @@
import { Injectable, Logger } from '@nestjs/common';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { Logger } from '@nestjs/common';
import { BillingService } from 'src/engine/core-modules/billing/billing.service';
import { UserWorkspaceService } from 'src/engine/core-modules/user-workspace/user-workspace.service';
import { StripeService } from 'src/engine/core-modules/billing/stripe/stripe.service';
import { Processor } from 'src/engine/integrations/message-queue/decorators/processor.decorator';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { Process } from 'src/engine/integrations/message-queue/decorators/process.decorator';
export type UpdateSubscriptionJobData = { workspaceId: string };
@Injectable()
export class UpdateSubscriptionJob
implements MessageQueueJob<UpdateSubscriptionJobData>
{
@Processor(MessageQueue.billingQueue)
export class UpdateSubscriptionJob {
protected readonly logger = new Logger(UpdateSubscriptionJob.name);
constructor(
private readonly billingService: BillingService,
private readonly userWorkspaceService: UserWorkspaceService,
private readonly stripeService: StripeService,
) {}
@Process(UpdateSubscriptionJob.name)
async handle(data: UpdateSubscriptionJobData): Promise<void> {
const workspaceMembersCount =
await this.userWorkspaceService.getWorkspaceMemberCount(data.workspaceId);

5
packages/twenty-server/src/engine/core-modules/billing/listeners/billing-workspace-member.listener.ts
View File

@ -1,4 +1,4 @@
import { Inject, Injectable } from '@nestjs/common';
import { Injectable } from '@nestjs/common';
import { OnEvent } from '@nestjs/event-emitter';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
@ -10,11 +10,12 @@ import {
UpdateSubscriptionJobData,
} from 'src/engine/core-modules/billing/jobs/update-subscription.job';
import { EnvironmentService } from 'src/engine/integrations/environment/environment.service';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
@Injectable()
export class BillingWorkspaceMemberListener {
constructor(
@Inject(MessageQueue.billingQueue)
@InjectMessageQueue(MessageQueue.billingQueue)
private readonly messageQueueService: MessageQueueService,
private readonly environmentService: EnvironmentService,
) {}

4
packages/twenty-server/src/engine/core-modules/core-engine.module.ts
View File

@ -10,7 +10,6 @@ import { TimelineMessagingModule } from 'src/engine/core-modules/messaging/timel
import { TimelineCalendarEventModule } from 'src/engine/core-modules/calendar/timeline-calendar-event.module';
import { BillingModule } from 'src/engine/core-modules/billing/billing.module';
import { HealthModule } from 'src/engine/core-modules/health/health.module';
import { TwentyORMModule } from 'src/engine/twenty-orm/twenty-orm.module';
import { PostgresCredentialsModule } from 'src/engine/core-modules/postgres-credentials/postgres-credentials.module';
import { AnalyticsModule } from './analytics/analytics.module';
@ -19,9 +18,6 @@ import { ClientConfigModule } from './client-config/client-config.module';
@Module({
imports: [
TwentyORMModule.register({
workspaceEntities: ['dist/src/**/*.workspace-entity{.ts,.js}'],
}),
HealthModule,
AnalyticsModule,
AuthModule,

15
packages/twenty-server/src/engine/core-modules/workspace/handle-workspace-member-deleted.job.ts
View File

@ -1,19 +1,18 @@
import { Injectable } from '@nestjs/common';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { WorkspaceService } from 'src/engine/core-modules/workspace/services/workspace.service';
import { Processor } from 'src/engine/integrations/message-queue/decorators/processor.decorator';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { Process } from 'src/engine/integrations/message-queue/decorators/process.decorator';
export type HandleWorkspaceMemberDeletedJobData = {
workspaceId: string;
userId: string;
};
@Injectable()
export class HandleWorkspaceMemberDeletedJob
implements MessageQueueJob<HandleWorkspaceMemberDeletedJobData>
{
@Processor(MessageQueue.workspaceQueue)
export class HandleWorkspaceMemberDeletedJob {
constructor(private readonly workspaceService: WorkspaceService) {}
@Process(HandleWorkspaceMemberDeletedJob.name)
async handle(data: HandleWorkspaceMemberDeletedJobData): Promise<void> {
const { workspaceId, userId } = data;

5
packages/twenty-server/src/engine/core-modules/workspace/workspace-workspace-member.listener.ts
View File

@ -1,4 +1,4 @@
import { Inject, Injectable } from '@nestjs/common';
import { Injectable } from '@nestjs/common';
import { OnEvent } from '@nestjs/event-emitter';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
@ -9,11 +9,12 @@ import {
HandleWorkspaceMemberDeletedJob,
HandleWorkspaceMemberDeletedJobData,
} from 'src/engine/core-modules/workspace/handle-workspace-member-deleted.job';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
@Injectable()
export class WorkspaceWorkspaceMemberListener {
constructor(
@Inject(MessageQueue.workspaceQueue)
@InjectMessageQueue(MessageQueue.workspaceQueue)
private readonly messageQueueService: MessageQueueService,
) {}

12
packages/twenty-server/src/engine/integrations/email/email-sender.job.ts
View File

@ -1,15 +1,15 @@
import { Injectable } from '@nestjs/common';
import { SendMailOptions } from 'nodemailer';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { EmailSenderService } from 'src/engine/integrations/email/email-sender.service';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { Processor } from 'src/engine/integrations/message-queue/decorators/processor.decorator';
import { Process } from 'src/engine/integrations/message-queue/decorators/process.decorator';
@Injectable()
export class EmailSenderJob implements MessageQueueJob<SendMailOptions> {
@Processor(MessageQueue.emailQueue)
export class EmailSenderJob {
constructor(private readonly emailSenderService: EmailSenderService) {}
@Process(EmailSenderJob.name)
async handle(data: SendMailOptions): Promise<void> {
await this.emailSenderService.send(data);
}

5
packages/twenty-server/src/engine/integrations/email/email.service.ts
View File

@ -1,15 +1,16 @@
import { Inject, Injectable } from '@nestjs/common';
import { Injectable } from '@nestjs/common';
import { SendMailOptions } from 'nodemailer';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { EmailSenderJob } from 'src/engine/integrations/email/email-sender.job';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
@Injectable()
export class EmailService {
constructor(
@Inject(MessageQueue.emailQueue)
@InjectMessageQueue(MessageQueue.emailQueue)
private readonly messageQueueService: MessageQueueService,
) {}

2
packages/twenty-server/src/engine/integrations/integrations.module.ts
View File

@ -30,7 +30,7 @@ import { MessageQueueModule } from './message-queue/message-queue.module';
useFactory: loggerModuleFactory,
inject: [EnvironmentService],
}),
MessageQueueModule.forRoot({
MessageQueueModule.registerAsync({
useFactory: messageQueueModuleFactory,
inject: [EnvironmentService],
}),

5
packages/twenty-server/src/engine/integrations/message-queue/decorators/message-queue.decorator.ts
View File

@ -1,7 +1,8 @@
import { Inject } from '@nestjs/common';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { getQueueToken } from 'src/engine/integrations/message-queue/utils/get-queue-token.util';
export const InjectMessageQueue = (messageQueueName: MessageQueue) => {
return Inject(messageQueueName);
export const InjectMessageQueue = (queueName: MessageQueue) => {
return Inject(getQueueToken(queueName));
};

21
packages/twenty-server/src/engine/integrations/message-queue/decorators/process.decorator.ts Normal file
View File

@ -0,0 +1,21 @@
import { SetMetadata } from '@nestjs/common';
import { isString } from '@nestjs/common/utils/shared.utils';
import { PROCESS_METADATA } from 'src/engine/integrations/message-queue/message-queue.constants';
export interface MessageQueueProcessOptions {
jobName: string;
concurrency?: number;
}
export function Process(jobName: string): MethodDecorator;
export function Process(options: MessageQueueProcessOptions): MethodDecorator;
export function Process(
nameOrOptions: string | MessageQueueProcessOptions,
): MethodDecorator {
const options = isString(nameOrOptions)
? { jobName: nameOrOptions }
: nameOrOptions;
return SetMetadata(PROCESS_METADATA, options || {});
}

69
packages/twenty-server/src/engine/integrations/message-queue/decorators/processor.decorator.ts Normal file
View File

@ -0,0 +1,69 @@
import { Scope, SetMetadata } from '@nestjs/common';
import { SCOPE_OPTIONS_METADATA } from '@nestjs/common/constants';
import { MessageQueueWorkerOptions } from 'src/engine/integrations/message-queue/interfaces/message-queue-worker-options.interface';
import {
MessageQueue,
PROCESSOR_METADATA,
WORKER_METADATA,
} from 'src/engine/integrations/message-queue/message-queue.constants';
export interface MessageQueueProcessorOptions {
/**
* Specifies the name of the queue to subscribe to.
*/
queueName: MessageQueue;
/**
* Specifies the lifetime of an injected Processor.
*/
scope?: Scope;
}
/**
* Represents a worker that is able to process jobs from the queue.
* @param queueName name of the queue to process
*/
export function Processor(queueName: string): ClassDecorator;
/**
* Represents a worker that is able to process jobs from the queue.
* @param queueName name of the queue to process
* @param workerOptions additional worker options
*/
export function Processor(
queueName: string,
workerOptions: MessageQueueWorkerOptions,
): ClassDecorator;
/**
* Represents a worker that is able to process jobs from the queue.
* @param processorOptions processor options
*/
export function Processor(
processorOptions: MessageQueueProcessorOptions,
): ClassDecorator;
/**
* Represents a worker that is able to process jobs from the queue.
* @param processorOptions processor options (Nest-specific)
* @param workerOptions additional Bull worker options
*/
export function Processor(
processorOptions: MessageQueueProcessorOptions,
workerOptions: MessageQueueWorkerOptions,
): ClassDecorator;
export function Processor(
queueNameOrOptions?: string | MessageQueueProcessorOptions,
maybeWorkerOptions?: MessageQueueWorkerOptions,
): ClassDecorator {
const options =
queueNameOrOptions && typeof queueNameOrOptions === 'object'
? queueNameOrOptions
: { queueName: queueNameOrOptions };
// eslint-disable-next-line @typescript-eslint/ban-types
return (target: Function) => {
SetMetadata(SCOPE_OPTIONS_METADATA, options)(target);
SetMetadata(PROCESSOR_METADATA, options)(target);
maybeWorkerOptions &&
SetMetadata(WORKER_METADATA, maybeWorkerOptions)(target);
};
}

23
packages/twenty-server/src/engine/integrations/message-queue/drivers/bullmq.driver.ts
View File

@ -1,9 +1,14 @@
import { OnModuleDestroy } from '@nestjs/common';
import omitBy from 'lodash.omitby';
import { JobsOptions, Queue, QueueOptions, Worker } from 'bullmq';
import {
QueueCronJobOptions,
QueueJobOptions,
} from 'src/engine/integrations/message-queue/drivers/interfaces/job-options.interface';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { MessageQueueWorkerOptions } from 'src/engine/integrations/message-queue/interfaces/message-queue-worker-options.interface';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
@ -11,7 +16,7 @@ import { MessageQueueDriver } from './interfaces/message-queue-driver.interface'
export type BullMQDriverOptions = QueueOptions;
export class BullMQDriver implements MessageQueueDriver {
export class BullMQDriver implements MessageQueueDriver, OnModuleDestroy {
private queueMap: Record<MessageQueue, Queue> = {} as Record<
MessageQueue,
Queue
@ -27,7 +32,7 @@ export class BullMQDriver implements MessageQueueDriver {
this.queueMap[queueName] = new Queue(queueName, this.options);
}
async stop() {
async onModuleDestroy() {
const workers = Object.values(this.workerMap);
const queues = Object.values(this.queueMap);
@ -39,14 +44,22 @@ export class BullMQDriver implements MessageQueueDriver {
async work<T>(
queueName: MessageQueue,
handler: ({ data, id }: { data: T; id: string }) => Promise<void>,
handler: (job: MessageQueueJob<T>) => Promise<void>,
options?: MessageQueueWorkerOptions,
) {
const worker = new Worker(
queueName,
async (job) => {
await handler(job as { data: T; id: string });
// TODO: Correctly support for job.id
await handler({ data: job.data, id: job.id ?? '', name: job.name });
},
this.options,
omitBy(
{
...this.options,
concurrency: options?.concurrency,
},
(value) => value === undefined,
),
);
this.workerMap[queueName] = worker;

3
packages/twenty-server/src/engine/integrations/message-queue/drivers/interfaces/message-queue-driver.interface.ts
View File

@ -3,6 +3,7 @@ import {
QueueJobOptions,
} from 'src/engine/integrations/message-queue/drivers/interfaces/job-options.interface';
import { MessageQueueJobData } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { MessageQueueWorkerOptions } from 'src/engine/integrations/message-queue/interfaces/message-queue-worker-options.interface';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
@ -16,6 +17,7 @@ export interface MessageQueueDriver {
work<T extends MessageQueueJobData>(
queueName: MessageQueue,
handler: ({ data, id }: { data: T; id: string }) => Promise<void> | void,
options?: MessageQueueWorkerOptions,
);
addCron<T extends MessageQueueJobData | undefined>(
queueName: MessageQueue,
@ -24,6 +26,5 @@ export interface MessageQueueDriver {
options?: QueueCronJobOptions,
);
removeCron(queueName: MessageQueue, jobName: string, pattern?: string);
stop?(): Promise<void>;
register?(queueName: MessageQueue): void;
}

29
packages/twenty-server/src/engine/integrations/message-queue/drivers/pg-boss.driver.ts
View File

@ -1,9 +1,13 @@
import { OnModuleDestroy, OnModuleInit } from '@nestjs/common';
import PgBoss from 'pg-boss';
import {
QueueCronJobOptions,
QueueJobOptions,
} from 'src/engine/integrations/message-queue/drivers/interfaces/job-options.interface';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { MessageQueueWorkerOptions } from 'src/engine/integrations/message-queue/interfaces/message-queue-worker-options.interface';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
@ -13,26 +17,37 @@ export type PgBossDriverOptions = PgBoss.ConstructorOptions;
const DEFAULT_PG_BOSS_CRON_PATTERN_WHEN_NOT_PROVIDED = '*/1 * * * *';
export class PgBossDriver implements MessageQueueDriver {
export class PgBossDriver
implements MessageQueueDriver, OnModuleInit, OnModuleDestroy
{
private pgBoss: PgBoss;
constructor(options: PgBossDriverOptions) {
this.pgBoss = new PgBoss(options);
}
async stop() {
await this.pgBoss.stop();
async onModuleInit() {
await this.pgBoss.start();
}
async init(): Promise<void> {
await this.pgBoss.start();
async onModuleDestroy() {
await this.pgBoss.stop();
}
async work<T>(
queueName: string,
handler: ({ data, id }: { data: T; id: string }) => Promise<void>,
handler: (job: MessageQueueJob<T>) => Promise<void>,
options?: MessageQueueWorkerOptions,
) {
return this.pgBoss.work(`${queueName}.*`, handler);
return this.pgBoss.work<T>(
`${queueName}.*`,
{
teamConcurrency: options?.concurrency,
},
async (job) => {
await handler({ data: job.data, id: job.id, name: job.name });
},
);
}
async addCron<T>(

67
packages/twenty-server/src/engine/integrations/message-queue/drivers/sync.driver.ts
View File

@ -1,57 +1,66 @@
import { ModuleRef } from '@nestjs/core';
import { Logger } from '@nestjs/common';
import { MessageQueueDriver } from 'src/engine/integrations/message-queue/drivers/interfaces/message-queue-driver.interface';
import {
MessageQueueCronJobData,
MessageQueueJob,
MessageQueueJobData,
MessageQueueJob,
} from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { getJobClassName } from 'src/engine/integrations/message-queue/utils/get-job-class-name.util';
import { MessageQueueDriver } from './interfaces/message-queue-driver.interface';
export class SyncDriver implements MessageQueueDriver {
private readonly logger = new Logger(SyncDriver.name);
constructor(private readonly jobsModuleRef: ModuleRef) {}
private workersMap: {
[queueName: string]: (job: MessageQueueJob<any>) => Promise<void> | void;
} = {};
constructor() {}
async add<T extends MessageQueueJobData>(
_queueName: MessageQueue,
queueName: MessageQueue,
jobName: string,
data: T,
): Promise<void> {
const jobClassName = getJobClassName(jobName);
const job: MessageQueueJob<MessageQueueJobData> = this.jobsModuleRef.get(
jobClassName,
{ strict: false },
);
await job.handle(data);
await this.processJob(queueName, { id: '', name: jobName, data });
}
async addCron<T extends MessageQueueJobData | undefined>(
_queueName: MessageQueue,
queueName: MessageQueue,
jobName: string,
data: T,
): Promise<void> {
this.logger.log(`Running cron job with SyncDriver`);
const jobClassName = getJobClassName(jobName);
const job: MessageQueueCronJobData<MessageQueueJobData | undefined> =
this.jobsModuleRef.get(jobClassName, {
strict: true,
});
await job.handle(data);
await this.processJob(queueName, {
id: '',
name: jobName,
// TODO: Fix this type issue
data: data as any,
});
}
async removeCron(_queueName: MessageQueue, jobName: string) {
this.logger.log(`Removing '${jobName}' cron job with SyncDriver`);
return;
async removeCron(queueName: MessageQueue, jobName: string) {
this.logger.log(`Removing '${queueName}' cron job with SyncDriver`);
}
work() {
return;
work<T extends MessageQueueJobData>(
queueName: MessageQueue,
handler: (job: MessageQueueJob<T>) => Promise<void> | void,
) {
this.logger.log(`Registering handler for queue: ${queueName}`);
this.workersMap[queueName] = handler;
}
async processJob<T extends MessageQueueJobData>(
queueName: string,
job: MessageQueueJob<T>,
) {
const worker = this.workersMap[queueName];
if (worker) {
await worker(job);
} else {
this.logger.error(`No handler found for job: ${queueName}`);
}
}
}

2
packages/twenty-server/src/engine/integrations/message-queue/interfaces/index.ts
View File

@ -1 +1 @@
export * from './message-queue.interface';
export * from './message-queue-module-options.interface';

6
packages/twenty-server/src/engine/integrations/message-queue/interfaces/message-queue-job.interface.ts
View File

@ -1,5 +1,7 @@
export interface MessageQueueJob<T extends MessageQueueJobData | undefined> {
handle(data: T): Promise<void> | void;
export interface MessageQueueJob<T = any> {
id: string;
name: string;
data: T;
}
export interface MessageQueueCronJobData<

9
packages/twenty-server/src/engine/integrations/message-queue/interfaces/message-queue.interface.ts → packages/twenty-server/src/engine/integrations/message-queue/interfaces/message-queue-module-options.interface.ts
View File

@ -1,5 +1,3 @@
import { FactoryProvider, ModuleMetadata } from '@nestjs/common';
import { BullMQDriverOptions } from 'src/engine/integrations/message-queue/drivers/bullmq.driver';
import { PgBossDriverOptions } from 'src/engine/integrations/message-queue/drivers/pg-boss.driver';
@ -28,10 +26,3 @@ export type MessageQueueModuleOptions =
| PgBossDriverFactoryOptions
| BullMQDriverFactoryOptions
| SyncDriverFactoryOptions;
export type MessageQueueModuleAsyncOptions = {
useFactory: (
...args: any[]
) => MessageQueueModuleOptions | Promise<MessageQueueModuleOptions>;
} & Pick<ModuleMetadata, 'imports'> &
Pick<FactoryProvider, 'inject'>;

3
packages/twenty-server/src/engine/integrations/message-queue/interfaces/message-queue-worker-options.interface.ts Normal file
View File

@ -0,0 +1,3 @@
export interface MessageQueueWorkerOptions {
concurrency?: number;
}

24
packages/twenty-server/src/engine/integrations/message-queue/jobs.module.ts
View File

@ -23,9 +23,9 @@ import { CalendarMessagingParticipantJobModule } from 'src/modules/calendar-mess
import { CalendarCronJobModule } from 'src/modules/calendar/crons/jobs/calendar-cron-job.module';
import { CalendarJobModule } from 'src/modules/calendar/jobs/calendar-job.module';
import { AutoCompaniesAndContactsCreationJobModule } from 'src/modules/connected-account/auto-companies-and-contacts-creation/jobs/auto-companies-and-contacts-creation-job.module';
import { CalendarModule } from 'src/modules/calendar/calendar.module';
import { MessagingModule } from 'src/modules/messaging/messaging.module';
import { TimelineJobModule } from 'src/modules/timeline/jobs/timeline-job.module';
import { CalendarModule } from 'src/modules/calendar/calendar.module';
@Module({
imports: [
@ -39,11 +39,10 @@ import { TimelineJobModule } from 'src/modules/timeline/jobs/timeline-job.module
UserWorkspaceModule,
WorkspaceModule,
MessagingModule,
CalendarModule,
CalendarEventParticipantModule,
TimelineActivityModule,
StripeModule,
CalendarModule,
// JobsModules
WorkspaceQueryRunnerJobModule,
CalendarMessagingParticipantJobModule,
CalendarCronJobModule,
@ -52,20 +51,11 @@ import { TimelineJobModule } from 'src/modules/timeline/jobs/timeline-job.module
TimelineJobModule,
],
providers: [
{
provide: CleanInactiveWorkspaceJob.name,
useClass: CleanInactiveWorkspaceJob,
},
{ provide: EmailSenderJob.name, useClass: EmailSenderJob },
{
provide: DataSeedDemoWorkspaceJob.name,
useClass: DataSeedDemoWorkspaceJob,
},
{ provide: UpdateSubscriptionJob.name, useClass: UpdateSubscriptionJob },
{
provide: HandleWorkspaceMemberDeletedJob.name,
useClass: HandleWorkspaceMemberDeletedJob,
},
CleanInactiveWorkspaceJob,
EmailSenderJob,
DataSeedDemoWorkspaceJob,
UpdateSubscriptionJob,
HandleWorkspaceMemberDeletedJob,
],
})
export class JobsModule {

121
packages/twenty-server/src/engine/integrations/message-queue/message-queue-core.module.ts Normal file
View File

@ -0,0 +1,121 @@
import {
DynamicModule,
Global,
Logger,
Module,
Provider,
} from '@nestjs/common';
import { MessageQueueDriver } from 'src/engine/integrations/message-queue/drivers/interfaces/message-queue-driver.interface';
import { MessageQueueDriverType } from 'src/engine/integrations/message-queue/interfaces';
import {
MessageQueue,
QUEUE_DRIVER,
} from 'src/engine/integrations/message-queue/message-queue.constants';
import { PgBossDriver } from 'src/engine/integrations/message-queue/drivers/pg-boss.driver';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { BullMQDriver } from 'src/engine/integrations/message-queue/drivers/bullmq.driver';
import { SyncDriver } from 'src/engine/integrations/message-queue/drivers/sync.driver';
import { getQueueToken } from 'src/engine/integrations/message-queue/utils/get-queue-token.util';
import {
ASYNC_OPTIONS_TYPE,
ConfigurableModuleClass,
OPTIONS_TYPE,
} from 'src/engine/integrations/message-queue/message-queue.module-definition';
@Global()
@Module({})
export class MessageQueueCoreModule extends ConfigurableModuleClass {
private static readonly logger = new Logger(MessageQueueCoreModule.name);
static register(options: typeof OPTIONS_TYPE): DynamicModule {
const dynamicModule = super.register(options);
const driverProvider: Provider = {
provide: QUEUE_DRIVER,
useFactory: () => {
return this.createDriver(options);
},
};
const queueProviders = this.createQueueProviders();
return {
...dynamicModule,
providers: [
...(dynamicModule.providers ?? []),
driverProvider,
...queueProviders,
],
exports: [
...(dynamicModule.exports ?? []),
...Object.values(MessageQueue).map((queueName) =>
getQueueToken(queueName),
),
],
};
}
static registerAsync(options: typeof ASYNC_OPTIONS_TYPE): DynamicModule {
const dynamicModule = super.registerAsync(options);
const driverProvider: Provider = {
provide: QUEUE_DRIVER,
useFactory: async (...args: any[]) => {
const config = await options.useFactory!(...args);
return this.createDriver(config);
},
inject: options.inject || [],
};
const queueProviders = MessageQueueCoreModule.createQueueProviders();
return {
...dynamicModule,
providers: [
...(dynamicModule.providers ?? []),
driverProvider,
...queueProviders,
],
exports: [
...(dynamicModule.exports ?? []),
...Object.values(MessageQueue).map((queueName) =>
getQueueToken(queueName),
),
],
};
}
static async createDriver({ type, options }: typeof OPTIONS_TYPE) {
switch (type) {
case MessageQueueDriverType.PgBoss: {
return new PgBossDriver(options);
}
case MessageQueueDriverType.BullMQ: {
return new BullMQDriver(options);
}
case MessageQueueDriverType.Sync: {
return new SyncDriver();
}
default: {
this.logger.warn(
`Unsupported message queue driver type: ${type}. Using SyncDriver by default.`,
);
return new SyncDriver();
}
}
}
static createQueueProviders(): Provider[] {
return Object.values(MessageQueue).map((queueName) => ({
provide: getQueueToken(queueName),
useFactory: (driver: MessageQueueDriver) => {
return new MessageQueueService(driver, queueName);
},
inject: [QUEUE_DRIVER],
}));
}
}

54
packages/twenty-server/src/engine/integrations/message-queue/message-queue-metadata.accessor.ts Normal file
View File

@ -0,0 +1,54 @@
/* eslint-disable @typescript-eslint/ban-types */
import { Injectable, Type } from '@nestjs/common';
import { Reflector } from '@nestjs/core';
import { MessageQueueWorkerOptions } from 'src/engine/integrations/message-queue/interfaces/message-queue-worker-options.interface';
import { MessageQueueProcessOptions } from 'src/engine/integrations/message-queue/decorators/process.decorator';
import { MessageQueueProcessorOptions } from 'src/engine/integrations/message-queue/decorators/processor.decorator';
import {
PROCESSOR_METADATA,
PROCESS_METADATA,
WORKER_METADATA,
} from 'src/engine/integrations/message-queue/message-queue.constants';
@Injectable()
export class MessageQueueMetadataAccessor {
constructor(private readonly reflector: Reflector) {}
isProcessor(target: Type<any> | Function): boolean {
if (!target) {
return false;
}
return !!this.reflector.get(PROCESSOR_METADATA, target);
}
isProcess(target: Type<any> | Function): boolean {
if (!target) {
return false;
}
return !!this.reflector.get(PROCESS_METADATA, target);
}
getProcessorMetadata(
target: Type<any> | Function,
): MessageQueueProcessorOptions | undefined {
return this.reflector.get(PROCESSOR_METADATA, target);
}
getProcessMetadata(
target: Type<any> | Function,
): MessageQueueProcessOptions | undefined {
const metadata = this.reflector.get(PROCESS_METADATA, target);
return metadata;
}
getWorkerOptionsMetadata(
target: Type<any> | Function,
): MessageQueueWorkerOptions {
return this.reflector.get(WORKER_METADATA, target) ?? {};
}
}

6
packages/twenty-server/src/engine/integrations/message-queue/message-queue.constants.ts
View File

@ -1,4 +1,7 @@
export const QUEUE_DRIVER = Symbol('QUEUE_DRIVER');
export const PROCESSOR_METADATA = Symbol('message-queue:processor_metadata');
export const PROCESS_METADATA = Symbol('message-queue:process_metadata');
export const WORKER_METADATA = Symbol('bullmq:worker_metadata');
export const QUEUE_DRIVER = Symbol('message-queue:queue_driver');
export enum MessageQueue {
taskAssignedQueue = 'task-assigned-queue',
@ -12,4 +15,5 @@ export enum MessageQueue {
workspaceQueue = 'workspace-queue',
recordPositionBackfillQueue = 'record-position-backfill-queue',
entityEventsToDbQueue = 'entity-events-to-db-queue',
testQueue = 'test-queue',
}

209
packages/twenty-server/src/engine/integrations/message-queue/message-queue.explorer.ts Normal file
View File

@ -0,0 +1,209 @@
import { Injectable, Logger, OnModuleInit } from '@nestjs/common';
import {
DiscoveryService,
MetadataScanner,
ModuleRef,
createContextId,
} from '@nestjs/core';
import { Module } from '@nestjs/core/injector/module';
import { Injector } from '@nestjs/core/injector/injector';
import { InstanceWrapper } from '@nestjs/core/injector/instance-wrapper';
import { MessageQueueWorkerOptions } from 'src/engine/integrations/message-queue/interfaces/message-queue-worker-options.interface';
import {
MessageQueueJob,
MessageQueueJobData,
} from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { getQueueToken } from 'src/engine/integrations/message-queue/utils/get-queue-token.util';
import { ExceptionHandlerService } from 'src/engine/integrations/exception-handler/exception-handler.service';
import { shouldFilterException } from 'src/engine/utils/global-exception-handler.util';
import { MessageQueueMetadataAccessor } from './message-queue-metadata.accessor';
interface ProcessorGroup {
instance: object;
host: Module;
processMethodNames: string[];
isRequestScoped: boolean;
}
@Injectable()
export class MessageQueueExplorer implements OnModuleInit {
private readonly logger = new Logger('MessageQueueModule');
private readonly injector = new Injector();
constructor(
private readonly moduleRef: ModuleRef,
private readonly discoveryService: DiscoveryService,
private readonly metadataAccessor: MessageQueueMetadataAccessor,
private readonly metadataScanner: MetadataScanner,
private readonly exceptionHandlerService: ExceptionHandlerService,
) {}
onModuleInit() {
this.explore();
}
explore() {
const processors = this.discoveryService
.getProviders()
.filter((wrapper) =>
this.metadataAccessor.isProcessor(
!wrapper.metatype || wrapper.inject
? wrapper.instance?.constructor
: wrapper.metatype,
),
);
const groupedProcessors = this.groupProcessorsByQueueName(processors);
for (const [queueName, processorGroupCollection] of Object.entries(
groupedProcessors,
)) {
const queueToken = getQueueToken(queueName);
const messageQueueService = this.getQueueService(queueToken);
this.handleProcessorGroupCollection(
processorGroupCollection,
messageQueueService,
);
}
}
private groupProcessorsByQueueName(processors: InstanceWrapper[]) {
return processors.reduce(
(acc, wrapper) => {
const { instance, metatype } = wrapper;
const methodNames = this.metadataScanner.getAllMethodNames(instance);
const { queueName } =
this.metadataAccessor.getProcessorMetadata(
instance.constructor || metatype,
) ?? {};
const processMethodNames = methodNames.filter((name) =>
this.metadataAccessor.isProcess(instance[name]),
);
if (!queueName) {
this.logger.error(
`Processor ${wrapper.name} is missing queue name metadata`,
);
return acc;
}
if (!wrapper.host) {
this.logger.error(
`Processor ${wrapper.name} is missing host metadata`,
);
return acc;
}
if (!acc[queueName]) {
acc[queueName] = [];
}
acc[queueName].push({
instance,
host: wrapper.host,
processMethodNames,
isRequestScoped: !wrapper.isDependencyTreeStatic(),
});
return acc;
},
{} as Record<string, ProcessorGroup[]>,
);
}
private getQueueService(queueToken: string): MessageQueueService {
try {
return this.moduleRef.get<MessageQueueService>(queueToken, {
strict: false,
});
} catch (err) {
this.logger.error(`No queue found for token ${queueToken}`);
throw err;
}
}
private async handleProcessorGroupCollection(
processorGroupCollection: ProcessorGroup[],
queue: MessageQueueService,
options?: MessageQueueWorkerOptions,
) {
queue.work(async (job) => {
for (const processorGroup of processorGroupCollection) {
await this.handleProcessor(processorGroup, job);
}
}, options);
}
private async handleProcessor(
{ instance, host, processMethodNames, isRequestScoped }: ProcessorGroup,
job: MessageQueueJob<MessageQueueJobData>,
) {
const processMetadataCollection = new Map(
processMethodNames.map((name) => {
const metadata = this.metadataAccessor.getProcessMetadata(
instance[name],
);
return [name, metadata];
}),
);
if (isRequestScoped) {
const contextId = createContextId();
if (this.moduleRef.registerRequestByContextId) {
this.moduleRef.registerRequestByContextId(
{
// Add workspaceId to the request object
req: {
workspaceId: job.data.workspaceId,
},
},
contextId,
);
}
const contextInstance = await this.injector.loadPerContext(
instance,
host,
host.providers,
contextId,
);
await this.invokeProcessMethods(
contextInstance,
processMetadataCollection,
job,
);
} else {
await this.invokeProcessMethods(instance, processMetadataCollection, job);
}
}
private async invokeProcessMethods(
instance: object,
processMetadataCollection: Map<string, any>,
job: MessageQueueJob<MessageQueueJobData>,
) {
for (const [methodName, metadata] of processMetadataCollection) {
if (job.name === metadata?.jobName) {
try {
await instance[methodName].call(instance, job.data);
} catch (err) {
if (!shouldFilterException(err)) {
this.exceptionHandlerService.captureExceptions([err]);
}
throw err;
}
}
}
}
}

20
packages/twenty-server/src/engine/integrations/message-queue/message-queue.module-definition.ts Normal file
View File

@ -0,0 +1,20 @@
import { ConfigurableModuleBuilder } from '@nestjs/common';
import { MessageQueueModuleOptions } from 'src/engine/integrations/message-queue/interfaces';
export const {
ConfigurableModuleClass,
OPTIONS_TYPE,
ASYNC_OPTIONS_TYPE,
MODULE_OPTIONS_TOKEN,
} = new ConfigurableModuleBuilder<MessageQueueModuleOptions>()
.setExtras(
{
isGlobal: true,
},
(definition, extras) => ({
...definition,
global: extras.isGlobal,
}),
)
.build();

78
packages/twenty-server/src/engine/integrations/message-queue/message-queue.module.ts
View File

@ -1,62 +1,36 @@
import { DynamicModule, Global } from '@nestjs/common';
import { MessageQueueDriver } from 'src/engine/integrations/message-queue/drivers/interfaces/message-queue-driver.interface';
import { DynamicModule, Global, Module } from '@nestjs/common';
import { DiscoveryModule } from '@nestjs/core';
import { MessageQueueCoreModule } from 'src/engine/integrations/message-queue/message-queue-core.module';
import { MessageQueueMetadataAccessor } from 'src/engine/integrations/message-queue/message-queue-metadata.accessor';
import { MessageQueueExplorer } from 'src/engine/integrations/message-queue/message-queue.explorer';
import {
MessageQueueDriverType,
MessageQueueModuleAsyncOptions,
} from 'src/engine/integrations/message-queue/interfaces';
import {
MessageQueue,
QUEUE_DRIVER,
} from 'src/engine/integrations/message-queue/message-queue.constants';
import { PgBossDriver } from 'src/engine/integrations/message-queue/drivers/pg-boss.driver';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { BullMQDriver } from 'src/engine/integrations/message-queue/drivers/bullmq.driver';
import { SyncDriver } from 'src/engine/integrations/message-queue/drivers/sync.driver';
import { JobsModule } from 'src/engine/integrations/message-queue/jobs.module';
ASYNC_OPTIONS_TYPE,
OPTIONS_TYPE,
} from 'src/engine/integrations/message-queue/message-queue.module-definition';
@Global()
@Module({})
export class MessageQueueModule {
static forRoot(options: MessageQueueModuleAsyncOptions): DynamicModule {
const providers = [
...Object.values(MessageQueue).map((queue) => ({
provide: queue,
useFactory: (driver: MessageQueueDriver) => {
return new MessageQueueService(driver, queue);
},
inject: [QUEUE_DRIVER],
})),
{
provide: QUEUE_DRIVER,
useFactory: async (...args: any[]) => {
const config = await options.useFactory(...args);
switch (config.type) {
case MessageQueueDriverType.PgBoss: {
const boss = new PgBossDriver(config.options);
await boss.init();
return boss;
}
case MessageQueueDriverType.BullMQ: {
return new BullMQDriver(config.options);
}
default: {
return new SyncDriver(JobsModule.moduleRef);
}
}
},
inject: options.inject || [],
},
];
static register(options: typeof OPTIONS_TYPE): DynamicModule {
return {
module: MessageQueueModule,
imports: [JobsModule, ...(options.imports || [])],
providers,
exports: Object.values(MessageQueue),
imports: [MessageQueueCoreModule.register(options)],
};
}
static registerExplorer(): DynamicModule {
return {
module: MessageQueueModule,
imports: [DiscoveryModule],
providers: [MessageQueueExplorer, MessageQueueMetadataAccessor],
};
}
static registerAsync(options: typeof ASYNC_OPTIONS_TYPE): DynamicModule {
return {
module: MessageQueueModule,
imports: [MessageQueueCoreModule.registerAsync(options)],
};
}
}

21
packages/twenty-server/src/engine/integrations/message-queue/services/message-queue.service.ts
View File

@ -1,11 +1,15 @@
import { Inject, Injectable, OnModuleDestroy } from '@nestjs/common';
import { Inject, Injectable } from '@nestjs/common';
import {
QueueCronJobOptions,
QueueJobOptions,
} from 'src/engine/integrations/message-queue/drivers/interfaces/job-options.interface';
import { MessageQueueDriver } from 'src/engine/integrations/message-queue/drivers/interfaces/message-queue-driver.interface';
import { MessageQueueJobData } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import {
MessageQueueJobData,
MessageQueueJob,
} from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { MessageQueueWorkerOptions } from 'src/engine/integrations/message-queue/interfaces/message-queue-worker-options.interface';
import {
MessageQueue,
@ -13,7 +17,7 @@ import {
} from 'src/engine/integrations/message-queue/message-queue.constants';
@Injectable()
export class MessageQueueService implements OnModuleDestroy {
export class MessageQueueService {
constructor(
@Inject(QUEUE_DRIVER) protected driver: MessageQueueDriver,
protected queueName: MessageQueue,
@ -23,12 +27,6 @@ export class MessageQueueService implements OnModuleDestroy {
}
}
async onModuleDestroy() {
if (typeof this.driver.stop === 'function') {
await this.driver.stop();
}
}
add<T extends MessageQueueJobData>(
jobName: string,
data: T,
@ -50,8 +48,9 @@ export class MessageQueueService implements OnModuleDestroy {
}
work<T extends MessageQueueJobData>(
handler: ({ data, id }: { data: T; id: string }) => Promise<void> | void,
handler: (job: MessageQueueJob<T>) => Promise<void> | void,
options?: MessageQueueWorkerOptions,
) {
return this.driver.work(this.queueName, handler);
return this.driver.work(this.queueName, handler, options);
}
}

2
packages/twenty-server/src/engine/integrations/message-queue/utils/get-queue-token.util.ts Normal file
View File

@ -0,0 +1,2 @@
export const getQueueToken = (queueName: string) =>
`MESSAGE_QUEUE_${queueName}`;

1
packages/twenty-server/src/engine/middlewares/graphql-hydrate-request-from-token.middleware.ts
View File

@ -55,6 +55,7 @@ export class GraphQLHydrateRequestFromTokenMiddleware
req.user = data.user;
req.workspace = data.workspace;
req.workspaceId = data.workspace.id;
req.cacheVersion = cacheVersion;
} catch (error) {
res.writeHead(200, { 'Content-Type': 'application/json' });

8
packages/twenty-server/src/engine/twenty-orm/factories/scoped-workspace-datasource.factory.ts
View File

@ -3,7 +3,6 @@ import { REQUEST } from '@nestjs/core';
import { EntitySchema } from 'typeorm';
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
import { WorkspaceDatasourceFactory } from 'src/engine/twenty-orm/factories/workspace-datasource.factory';
@Injectable({ scope: Scope.REQUEST })
@ -14,12 +13,13 @@ export class ScopedWorkspaceDatasourceFactory {
) {}
public async create(entities: EntitySchema[]) {
const workspace: Workspace | undefined = this.request['req']?.['workspace'];
const workspaceId: string | undefined =
this.request['req']?.['workspaceId'];
if (!workspace) {
if (!workspaceId) {
return null;
}
return this.workspaceDataSourceFactory.create(entities, workspace.id);
return this.workspaceDataSourceFactory.create(entities, workspaceId);
}
}

9
packages/twenty-server/src/engine/twenty-orm/twenty-orm-core.module.ts
View File

@ -7,10 +7,6 @@ import {
Provider,
Type,
} from '@nestjs/common';
import {
ConfigurableModuleClass,
MODULE_OPTIONS_TOKEN,
} from '@nestjs/common/cache/cache.module-definition';
import { importClassesFromDirectories } from 'typeorm/util/DirectoryExportedClassesLoader';
import { Logger as TypeORMLogger } from 'typeorm/logger/Logger';
@ -30,6 +26,10 @@ import { ScopedWorkspaceDatasourceFactory } from 'src/engine/twenty-orm/factorie
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
import { splitClassesAndStrings } from 'src/engine/twenty-orm/utils/split-classes-and-strings.util';
import { CustomWorkspaceEntity } from 'src/engine/twenty-orm/custom.workspace-entity';
import {
ConfigurableModuleClass,
MODULE_OPTIONS_TOKEN,
} from 'src/engine/twenty-orm/twenty-orm.module-definition';
@Global()
@Module({
@ -46,7 +46,6 @@ export class TwentyORMCoreModule
static register(options: TwentyORMOptions): DynamicModule {
const dynamicModule = super.register(options);
console.log('register', options);
const providers: Provider[] = [
{
provide: TWENTY_ORM_WORKSPACE_DATASOURCE,

3
packages/twenty-server/src/engine/twenty-orm/twenty-orm.module.ts
View File

@ -1,5 +1,4 @@
import { DynamicModule, Global, Module } from '@nestjs/common';
import { ConfigurableModuleClass } from '@nestjs/common/cache/cache.module-definition';
import { EntityClassOrSchema } from '@nestjs/typeorm/dist/interfaces/entity-class-or-schema.type';
import {
@ -12,7 +11,7 @@ import { TwentyORMCoreModule } from 'src/engine/twenty-orm/twenty-orm-core.modul
@Global()
@Module({})
export class TwentyORMModule extends ConfigurableModuleClass {
export class TwentyORMModule {
static register(options: TwentyORMOptions): DynamicModule {
return {
module: TwentyORMModule,

5
packages/twenty-server/src/engine/workspace-manager/workspace-cleaner/commands/clean-inactive-workspaces.command.ts
View File

@ -1,7 +1,6 @@
import { Inject } from '@nestjs/common';
import { Command, CommandRunner, Option } from 'nest-commander';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { CleanInactiveWorkspaceJob } from 'src/engine/workspace-manager/workspace-cleaner/crons/clean-inactive-workspace.job';
@ -16,7 +15,7 @@ export type CleanInactiveWorkspacesCommandOptions = {
})
export class CleanInactiveWorkspacesCommand extends CommandRunner {
constructor(
@Inject(MessageQueue.taskAssignedQueue)
@InjectMessageQueue(MessageQueue.cronQueue)
private readonly messageQueueService: MessageQueueService,
) {
super();

5
packages/twenty-server/src/engine/workspace-manager/workspace-cleaner/commands/start-clean-inactive-workspaces.cron.command.ts
View File

@ -1,7 +1,6 @@
import { Inject } from '@nestjs/common';
import { Command, CommandRunner } from 'nest-commander';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { cleanInactiveWorkspaceCronPattern } from 'src/engine/workspace-manager/workspace-cleaner/crons/clean-inactive-workspace.cron.pattern';
@ -13,7 +12,7 @@ import { CleanInactiveWorkspaceJob } from 'src/engine/workspace-manager/workspac
})
export class StartCleanInactiveWorkspacesCronCommand extends CommandRunner {
constructor(
@Inject(MessageQueue.cronQueue)
@InjectMessageQueue(MessageQueue.cronQueue)
private readonly messageQueueService: MessageQueueService,
) {
super();

5
packages/twenty-server/src/engine/workspace-manager/workspace-cleaner/commands/stop-clean-inactive-workspaces.cron.command.ts
View File

@ -1,7 +1,6 @@
import { Inject } from '@nestjs/common';
import { Command, CommandRunner } from 'nest-commander';
import { InjectMessageQueue } from 'src/engine/integrations/message-queue/decorators/message-queue.decorator';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { cleanInactiveWorkspaceCronPattern } from 'src/engine/workspace-manager/workspace-cleaner/crons/clean-inactive-workspace.cron.pattern';
@ -13,7 +12,7 @@ import { CleanInactiveWorkspaceJob } from 'src/engine/workspace-manager/workspac
})
export class StopCleanInactiveWorkspacesCronCommand extends CommandRunner {
constructor(
@Inject(MessageQueue.cronQueue)
@InjectMessageQueue(MessageQueue.cronQueue)
private readonly messageQueueService: MessageQueueService,
) {
super();

14
packages/twenty-server/src/engine/workspace-manager/workspace-cleaner/crons/clean-inactive-workspace.job.ts
View File

@ -1,4 +1,4 @@
import { Injectable, Logger } from '@nestjs/common';
import { Logger } from '@nestjs/common';
import { render } from '@react-email/render';
import { In } from 'typeorm';
@ -7,8 +7,6 @@ import {
DeleteInactiveWorkspaceEmail,
} from 'twenty-emails';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { ObjectMetadataService } from 'src/engine/metadata-modules/object-metadata/object-metadata.service';
import { DataSourceService } from 'src/engine/metadata-modules/data-source/data-source.service';
import { TypeORMService } from 'src/database/typeorm/typeorm.service';
@ -20,6 +18,9 @@ import { ObjectMetadataEntity } from 'src/engine/metadata-modules/object-metadat
import { computeObjectTargetTable } from 'src/engine/utils/compute-object-target-table.util';
import { CleanInactiveWorkspacesCommandOptions } from 'src/engine/workspace-manager/workspace-cleaner/commands/clean-inactive-workspaces.command';
import { getDryRunLogHeader } from 'src/utils/get-dry-run-log-header';
import { Processor } from 'src/engine/integrations/message-queue/decorators/processor.decorator';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { Process } from 'src/engine/integrations/message-queue/decorators/process.decorator';
const MILLISECONDS_IN_ONE_DAY = 1000 * 3600 * 24;
@ -28,10 +29,8 @@ type WorkspaceToDeleteData = {
daysSinceInactive: number;
};
@Injectable()
export class CleanInactiveWorkspaceJob
implements MessageQueueJob<CleanInactiveWorkspacesCommandOptions>
{
@Processor(MessageQueue.cronQueue)
export class CleanInactiveWorkspaceJob {
private readonly logger = new Logger(CleanInactiveWorkspaceJob.name);
private readonly inactiveDaysBeforeDelete;
private readonly inactiveDaysBeforeEmail;
@ -193,6 +192,7 @@ export class CleanInactiveWorkspaceJob
});
}
@Process(CleanInactiveWorkspaceJob.name)
async handle(data: CleanInactiveWorkspacesCommandOptions): Promise<void> {
const isDryRun = data.dryRun || false;