Rework messaging modules (#5710)

In this PR, I'm refactoring the messaging module into smaller pieces
that have **ONE** responsibility: import messages, clean messages,
handle message participant creation, instead of having ~30 modules (1
per service, jobs, cron, ...). This is mandatory to start introducing
drivers (gmails, office365, ...) IMO. It is too difficult to enforce
common interfaces as we have too many interfaces (30 modules...). All
modules should not be exposed

Right now, we have services that are almost functions:
do-that-and-this.service.ts / do-that-and-this.module.ts
I believe we should have something more organized at a high level and it
does not matter that much if we have a bit of code duplicates.

Note that the proposal is not fully implemented in the current PR that
has only focused on messaging folder (biggest part)

Here is the high level proposal:
- connected-account: token-refresher
- blocklist
- messaging: message-importer, message-cleaner, message-participants,
... (right now I'm keeping a big messaging-common but this will
disappear see below)
- calendar: calendar-importer, calendar-cleaner, ...

Consequences:
1) It's OK to re-implement several times some things. Example:
- error handling in connected-account, messaging, and calendar instead
of trying to unify. They are actually different error handling. The only
things that might be in common is the GmailError => CommonError parsing
and I'm not even sure it makes a lot of sense as these 3 apis might have
different format actually
- auto-creation. Calendar and Messaging could actually have different
rules

2) **We should not have circular dependencies:** 
- I believe this was the reason why we had so many modules, to be able
to cherry pick the one we wanted to avoid circular deps. This is not the
right approach IMO, we need architect the whole messaging by defining
high level blocks that won't have circular dependencies by design. If we
encounter one, we should rethink and break the block in a way that makes
sense.
- ex: connected-account.resolver is not in the same module as
token-refresher. ==> connected-account.resolver => message-importer (as
we trigger full sync job when we connect an account) => token-refresher
(as we refresh token on message import).

connected-account.resolver and token-refresher both in connected-account
folder but should be in different modules. Otherwise it's a circular
dependency. It does not mean that we should create 1 module per service
as it was done before

In a nutshell: The code needs to be thought in term of reponsibilities
and in a way that enforce high level interfaces (and avoid circular
dependencies)

Bonus: As you can see, this code is also removing a lot of code because
of the removal of many .module.ts (also because I'm removing the sync
scripts v2 feature flag end removing old code)
Bonus: I have prefixed services name with Messaging to improve dev xp.
GmailErrorHandler could be different between MessagingGmailErrorHandler
and CalendarGmailErrorHandler for instance
This commit is contained in:
Charles Bochet
2024-06-03 11:16:05 +02:00
committed by GitHub
parent c4b6b1e076
commit eab8deb211
121 changed files with 690 additions and 2065 deletions

View File

@ -7,7 +7,6 @@ import { CalendarCronCommandsModule } from 'src/modules/calendar/crons/commands/
import { AppModule } from 'src/app.module'; import { AppModule } from 'src/app.module';
import { WorkspaceMigrationRunnerCommandsModule } from 'src/engine/workspace-manager/workspace-migration-runner/commands/workspace-sync-metadata-commands.module'; import { WorkspaceMigrationRunnerCommandsModule } from 'src/engine/workspace-manager/workspace-migration-runner/commands/workspace-sync-metadata-commands.module';
import { WorkspaceSyncMetadataCommandsModule } from 'src/engine/workspace-manager/workspace-sync-metadata/commands/workspace-sync-metadata-commands.module'; import { WorkspaceSyncMetadataCommandsModule } from 'src/engine/workspace-manager/workspace-sync-metadata/commands/workspace-sync-metadata-commands.module';
import { MessagingCronCommandsModule } from 'src/modules/messaging/crons/commands/messaging-cron-commands.module';
import { CalendarCommandsModule } from 'src/modules/calendar/commands/calendar-commands.module'; import { CalendarCommandsModule } from 'src/modules/calendar/commands/calendar-commands.module';
@Module({ @Module({
@ -15,7 +14,7 @@ import { CalendarCommandsModule } from 'src/modules/calendar/commands/calendar-c
AppModule, AppModule,
WorkspaceSyncMetadataCommandsModule, WorkspaceSyncMetadataCommandsModule,
DatabaseCommandModule, DatabaseCommandModule,
MessagingCronCommandsModule, // MessagingCronCommandsModule,
CalendarCronCommandsModule, CalendarCronCommandsModule,
CalendarCommandsModule, CalendarCommandsModule,
WorkspaceCleanerModule, WorkspaceCleanerModule,

View File

@ -12,7 +12,7 @@ import { TypeORMService } from 'src/database/typeorm/typeorm.service';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity'; import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
import { WorkspaceCacheVersionService } from 'src/engine/metadata-modules/workspace-cache-version/workspace-cache-version.service'; import { WorkspaceCacheVersionService } from 'src/engine/metadata-modules/workspace-cache-version/workspace-cache-version.service';
import { ObjectMetadataEntity } from 'src/engine/metadata-modules/object-metadata/object-metadata.entity'; import { ObjectMetadataEntity } from 'src/engine/metadata-modules/object-metadata/object-metadata.entity';
import { MessageChannelSyncStatus } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; import { MessageChannelSyncStatus } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
interface UpdateMessageChannelSyncStatusEnumCommandOptions { interface UpdateMessageChannelSyncStatusEnumCommandOptions {
workspaceId?: string; workspaceId?: string;

View File

@ -40,11 +40,6 @@ export const seedFeatureFlags = async (
workspaceId: workspaceId, workspaceId: workspaceId,
value: true, value: true,
}, },
{
key: FeatureFlagKeys.IsGmailSyncV2Enabled,
workspaceId: workspaceId,
value: true,
},
{ {
key: FeatureFlagKeys.IsContactCreationForSentAndReceivedEmailsEnabled, key: FeatureFlagKeys.IsContactCreationForSentAndReceivedEmailsEnabled,
workspaceId: workspaceId, workspaceId: workspaceId,

View File

@ -1,7 +1,7 @@
import { EntityManager } from 'typeorm'; import { EntityManager } from 'typeorm';
import { DEV_SEED_CONNECTED_ACCOUNT_IDS } from 'src/database/typeorm-seeds/workspace/connected-account'; import { DEV_SEED_CONNECTED_ACCOUNT_IDS } from 'src/database/typeorm-seeds/workspace/connected-account';
import { MessageChannelSyncSubStatus } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; import { MessageChannelSyncSubStatus } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
const tableName = 'messageChannel'; const tableName = 'messageChannel';

View File

@ -1,5 +1,3 @@
import { MessageFindManyPreQueryHook } from 'src/modules/messaging/query-hooks/message/message-find-many.pre-query.hook';
import { MessageFindOnePreQueryHook } from 'src/modules/messaging/query-hooks/message/message-find-one.pre-query-hook';
import { WorkspaceQueryHook } from 'src/engine/api/graphql/workspace-query-runner/workspace-pre-query-hook/types/workspace-query-hook.type'; import { WorkspaceQueryHook } from 'src/engine/api/graphql/workspace-query-runner/workspace-pre-query-hook/types/workspace-query-hook.type';
import { CalendarEventFindManyPreQueryHook } from 'src/modules/calendar/query-hooks/calendar-event/calendar-event-find-many.pre-query.hook'; import { CalendarEventFindManyPreQueryHook } from 'src/modules/calendar/query-hooks/calendar-event/calendar-event-find-many.pre-query.hook';
import { CalendarEventFindOnePreQueryHook } from 'src/modules/calendar/query-hooks/calendar-event/calendar-event-find-one.pre-query-hook'; import { CalendarEventFindOnePreQueryHook } from 'src/modules/calendar/query-hooks/calendar-event/calendar-event-find-one.pre-query-hook';
@ -8,6 +6,8 @@ import { BlocklistUpdateManyPreQueryHook } from 'src/modules/connected-account/q
import { BlocklistUpdateOnePreQueryHook } from 'src/modules/connected-account/query-hooks/blocklist/blocklist-update-one.pre-query.hook'; import { BlocklistUpdateOnePreQueryHook } from 'src/modules/connected-account/query-hooks/blocklist/blocklist-update-one.pre-query.hook';
import { WorkspaceMemberDeleteOnePreQueryHook } from 'src/modules/workspace-member/query-hooks/workspace-member-delete-one.pre-query.hook'; import { WorkspaceMemberDeleteOnePreQueryHook } from 'src/modules/workspace-member/query-hooks/workspace-member-delete-one.pre-query.hook';
import { WorkspaceMemberDeleteManyPreQueryHook } from 'src/modules/workspace-member/query-hooks/workspace-member-delete-many.pre-query.hook'; import { WorkspaceMemberDeleteManyPreQueryHook } from 'src/modules/workspace-member/query-hooks/workspace-member-delete-many.pre-query.hook';
import { MessageFindManyPreQueryHook } from 'src/modules/messaging/common/query-hooks/message/message-find-many.pre-query.hook';
import { MessageFindOnePreQueryHook } from 'src/modules/messaging/common/query-hooks/message/message-find-one.pre-query-hook';
// TODO: move to a decorator // TODO: move to a decorator
export const workspacePreQueryHooks: WorkspaceQueryHook = { export const workspacePreQueryHooks: WorkspaceQueryHook = {

View File

@ -1,9 +1,9 @@
import { Module } from '@nestjs/common'; import { Module } from '@nestjs/common';
import { MessagingQueryHookModule } from 'src/modules/messaging/query-hooks/messaging-query-hook.module';
import { WorkspacePreQueryHookService } from 'src/engine/api/graphql/workspace-query-runner/workspace-pre-query-hook/workspace-pre-query-hook.service'; import { WorkspacePreQueryHookService } from 'src/engine/api/graphql/workspace-query-runner/workspace-pre-query-hook/workspace-pre-query-hook.service';
import { CalendarQueryHookModule } from 'src/modules/calendar/query-hooks/calendar-query-hook.module'; import { CalendarQueryHookModule } from 'src/modules/calendar/query-hooks/calendar-query-hook.module';
import { ConnectedAccountQueryHookModule } from 'src/modules/connected-account/query-hooks/connected-account-query-hook.module'; import { ConnectedAccountQueryHookModule } from 'src/modules/connected-account/query-hooks/connected-account-query-hook.module';
import { MessagingQueryHookModule } from 'src/modules/messaging/common/query-hooks/messaging-query-hook.module';
import { WorkspaceMemberQueryHookModule } from 'src/modules/workspace-member/query-hooks/workspace-member-query-hook.module'; import { WorkspaceMemberQueryHookModule } from 'src/modules/workspace-member/query-hooks/workspace-member-query-hook.module';
@Module({ @Module({

View File

@ -25,8 +25,8 @@ import { MicrosoftAuthController } from 'src/engine/core-modules/auth/controller
import { AppTokenService } from 'src/engine/core-modules/app-token/services/app-token.service'; import { AppTokenService } from 'src/engine/core-modules/app-token/services/app-token.service';
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module'; import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { CalendarChannelWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-channel.workspace-entity'; import { CalendarChannelWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-channel.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { AuthResolver } from './auth.resolver'; import { AuthResolver } from './auth.resolver';

View File

@ -1,11 +1,9 @@
import { Injectable, Inject } from '@nestjs/common'; import { Injectable, Inject } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository, EntityManager } from 'typeorm'; import { EntityManager } from 'typeorm';
import { v4 } from 'uuid'; import { v4 } from 'uuid';
import { TypeORMService } from 'src/database/typeorm/typeorm.service'; import { TypeORMService } from 'src/database/typeorm/typeorm.service';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { EnvironmentService } from 'src/engine/integrations/environment/environment.service'; import { EnvironmentService } from 'src/engine/integrations/environment/environment.service';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants'; import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service'; import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
@ -25,16 +23,16 @@ import {
ConnectedAccountWorkspaceEntity, ConnectedAccountWorkspaceEntity,
ConnectedAccountProvider, ConnectedAccountProvider,
} from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository'; import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { import {
MessageChannelWorkspaceEntity, MessageChannelWorkspaceEntity,
MessageChannelType, MessageChannelType,
MessageChannelVisibility, MessageChannelVisibility,
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { import {
GmailFullMessageListFetchJobData, MessagingMessageListFetchJob,
GmailFullMessageListFetchJob, MessagingMessageListFetchJobData,
} from 'src/modules/messaging/jobs/gmail-full-message-list-fetch.job'; } from 'src/modules/messaging/message-import-manager/jobs/messaging-message-list-fetch.job';
@Injectable() @Injectable()
export class GoogleAPIsService { export class GoogleAPIsService {
@ -46,8 +44,6 @@ export class GoogleAPIsService {
@Inject(MessageQueue.calendarQueue) @Inject(MessageQueue.calendarQueue)
private readonly calendarQueueService: MessageQueueService, private readonly calendarQueueService: MessageQueueService,
private readonly environmentService: EnvironmentService, private readonly environmentService: EnvironmentService,
@InjectRepository(FeatureFlagEntity, 'core')
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity) @InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
private readonly connectedAccountRepository: ConnectedAccountRepository, private readonly connectedAccountRepository: ConnectedAccountRepository,
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity) @InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
@ -156,8 +152,8 @@ export class GoogleAPIsService {
isCalendarEnabled: boolean, isCalendarEnabled: boolean,
) { ) {
if (this.environmentService.get('MESSAGING_PROVIDER_GMAIL_ENABLED')) { if (this.environmentService.get('MESSAGING_PROVIDER_GMAIL_ENABLED')) {
await this.messageQueueService.add<GmailFullMessageListFetchJobData>( await this.messageQueueService.add<MessagingMessageListFetchJobData>(
GmailFullMessageListFetchJob.name, MessagingMessageListFetchJob.name,
{ {
workspaceId, workspaceId,
connectedAccountId, connectedAccountId,

View File

@ -21,7 +21,6 @@ export enum FeatureFlagKeys {
IsAirtableIntegrationEnabled = 'IS_AIRTABLE_INTEGRATION_ENABLED', IsAirtableIntegrationEnabled = 'IS_AIRTABLE_INTEGRATION_ENABLED',
IsPostgreSQLIntegrationEnabled = 'IS_POSTGRESQL_INTEGRATION_ENABLED', IsPostgreSQLIntegrationEnabled = 'IS_POSTGRESQL_INTEGRATION_ENABLED',
IsStripeIntegrationEnabled = 'IS_STRIPE_INTEGRATION_ENABLED', IsStripeIntegrationEnabled = 'IS_STRIPE_INTEGRATION_ENABLED',
IsGmailSyncV2Enabled = 'IS_GMAIL_SYNC_V2_ENABLED',
IsContactCreationForSentAndReceivedEmailsEnabled = 'IS_CONTACT_CREATION_FOR_SENT_AND_RECEIVED_EMAILS_ENABLED', IsContactCreationForSentAndReceivedEmailsEnabled = 'IS_CONTACT_CREATION_FOR_SENT_AND_RECEIVED_EMAILS_ENABLED',
} }

View File

@ -17,18 +17,14 @@ import { DataSourceModule } from 'src/engine/metadata-modules/data-source/data-s
import { ObjectMetadataModule } from 'src/engine/metadata-modules/object-metadata/object-metadata.module'; import { ObjectMetadataModule } from 'src/engine/metadata-modules/object-metadata/object-metadata.module';
import { CleanInactiveWorkspaceJob } from 'src/engine/workspace-manager/workspace-cleaner/crons/clean-inactive-workspace.job'; import { CleanInactiveWorkspaceJob } from 'src/engine/workspace-manager/workspace-cleaner/crons/clean-inactive-workspace.job';
import { CalendarEventParticipantModule } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.module'; import { CalendarEventParticipantModule } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.module';
import { GmailMessagesImportModule } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import.module';
import { GmailFullMessageListFetchModule } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch.module';
import { GmailPartialMessageListFetchModule } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-partial-message-list-fetch.module';
import { TimelineActivityModule } from 'src/modules/timeline/timeline-activity.module'; import { TimelineActivityModule } from 'src/modules/timeline/timeline-activity.module';
import { WorkspaceModule } from 'src/engine/core-modules/workspace/workspace.module'; import { WorkspaceModule } from 'src/engine/core-modules/workspace/workspace.module';
import { CalendarMessagingParticipantJobModule } from 'src/modules/calendar-messaging-participant/jobs/calendar-messaging-participant-job.module'; import { CalendarMessagingParticipantJobModule } from 'src/modules/calendar-messaging-participant/jobs/calendar-messaging-participant-job.module';
import { CalendarCronJobModule } from 'src/modules/calendar/crons/jobs/calendar-cron-job.module'; import { CalendarCronJobModule } from 'src/modules/calendar/crons/jobs/calendar-cron-job.module';
import { CalendarJobModule } from 'src/modules/calendar/jobs/calendar-job.module'; import { CalendarJobModule } from 'src/modules/calendar/jobs/calendar-job.module';
import { AutoCompaniesAndContactsCreationJobModule } from 'src/modules/connected-account/auto-companies-and-contacts-creation/jobs/auto-companies-and-contacts-creation-job.module'; import { AutoCompaniesAndContactsCreationJobModule } from 'src/modules/connected-account/auto-companies-and-contacts-creation/jobs/auto-companies-and-contacts-creation-job.module';
import { MessagingCronJobModule } from 'src/modules/messaging/crons/jobs/messaging-cron-job.module';
import { MessagingJobModule } from 'src/modules/messaging/jobs/messaging-job.module';
import { TimelineJobModule } from 'src/modules/timeline/jobs/timeline-job.module'; import { TimelineJobModule } from 'src/modules/timeline/jobs/timeline-job.module';
import { MessagingModule } from 'src/modules/messaging/messaging.module';
@Module({ @Module({
imports: [ imports: [
@ -41,9 +37,7 @@ import { TimelineJobModule } from 'src/modules/timeline/jobs/timeline-job.module
BillingModule, BillingModule,
UserWorkspaceModule, UserWorkspaceModule,
WorkspaceModule, WorkspaceModule,
GmailFullMessageListFetchModule, MessagingModule,
GmailMessagesImportModule,
GmailPartialMessageListFetchModule,
CalendarEventParticipantModule, CalendarEventParticipantModule,
TimelineActivityModule, TimelineActivityModule,
StripeModule, StripeModule,
@ -53,8 +47,6 @@ import { TimelineJobModule } from 'src/modules/timeline/jobs/timeline-job.module
CalendarCronJobModule, CalendarCronJobModule,
CalendarJobModule, CalendarJobModule,
AutoCompaniesAndContactsCreationJobModule, AutoCompaniesAndContactsCreationJobModule,
MessagingCronJobModule,
MessagingJobModule,
TimelineJobModule, TimelineJobModule,
], ],
providers: [ providers: [

View File

@ -7,15 +7,15 @@ import { BlocklistRepository } from 'src/modules/connected-account/repositories/
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository'; import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
import { AuditLogRepository } from 'src/modules/timeline/repositiories/audit-log.repository'; import { AuditLogRepository } from 'src/modules/timeline/repositiories/audit-log.repository';
import { TimelineActivityRepository } from 'src/modules/timeline/repositiories/timeline-activity.repository'; import { TimelineActivityRepository } from 'src/modules/timeline/repositiories/timeline-activity.repository';
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
import { MessageParticipantRepository } from 'src/modules/messaging/repositories/message-participant.repository';
import { MessageThreadRepository } from 'src/modules/messaging/repositories/message-thread.repository';
import { MessageRepository } from 'src/modules/messaging/repositories/message.repository';
import { PersonRepository } from 'src/modules/person/repositories/person.repository';
import { WorkspaceMemberRepository } from 'src/modules/workspace-member/repositories/workspace-member.repository'; import { WorkspaceMemberRepository } from 'src/modules/workspace-member/repositories/workspace-member.repository';
import { AttachmentRepository } from 'src/modules/attachment/repositories/attachment.repository'; import { AttachmentRepository } from 'src/modules/attachment/repositories/attachment.repository';
import { CommentRepository } from 'src/modules/activity/repositories/comment.repository'; import { CommentRepository } from 'src/modules/activity/repositories/comment.repository';
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { MessageParticipantRepository } from 'src/modules/messaging/common/repositories/message-participant.repository';
import { MessageThreadRepository } from 'src/modules/messaging/common/repositories/message-thread.repository';
import { MessageRepository } from 'src/modules/messaging/common/repositories/message.repository';
import { PersonRepository } from 'src/modules/person/repositories/person.repository';
export const metadataToRepositoryMapping = { export const metadataToRepositoryMapping = {
AuditLogWorkspaceEntity: AuditLogRepository, AuditLogWorkspaceEntity: AuditLogRepository,

View File

@ -58,7 +58,6 @@ export class AddStandardIdCommand extends CommandRunner {
IS_AIRTABLE_INTEGRATION_ENABLED: true, IS_AIRTABLE_INTEGRATION_ENABLED: true,
IS_POSTGRESQL_INTEGRATION_ENABLED: true, IS_POSTGRESQL_INTEGRATION_ENABLED: true,
IS_STRIPE_INTEGRATION_ENABLED: false, IS_STRIPE_INTEGRATION_ENABLED: false,
IS_GMAIL_SYNC_V2_ENABLED: true,
IS_CONTACT_CREATION_FOR_SENT_AND_RECEIVED_EMAILS_ENABLED: true, IS_CONTACT_CREATION_FOR_SENT_AND_RECEIVED_EMAILS_ENABLED: true,
}, },
); );
@ -74,7 +73,6 @@ export class AddStandardIdCommand extends CommandRunner {
IS_AIRTABLE_INTEGRATION_ENABLED: true, IS_AIRTABLE_INTEGRATION_ENABLED: true,
IS_POSTGRESQL_INTEGRATION_ENABLED: true, IS_POSTGRESQL_INTEGRATION_ENABLED: true,
IS_STRIPE_INTEGRATION_ENABLED: false, IS_STRIPE_INTEGRATION_ENABLED: false,
IS_GMAIL_SYNC_V2_ENABLED: true,
IS_CONTACT_CREATION_FOR_SENT_AND_RECEIVED_EMAILS_ENABLED: true, IS_CONTACT_CREATION_FOR_SENT_AND_RECEIVED_EMAILS_ENABLED: true,
}, },
); );

View File

@ -10,11 +10,6 @@ import { CommentWorkspaceEntity } from 'src/modules/activity/standard-objects/co
import { CompanyWorkspaceEntity } from 'src/modules/company/standard-objects/company.workspace-entity'; import { CompanyWorkspaceEntity } from 'src/modules/company/standard-objects/company.workspace-entity';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { FavoriteWorkspaceEntity } from 'src/modules/favorite/standard-objects/favorite.workspace-entity'; import { FavoriteWorkspaceEntity } from 'src/modules/favorite/standard-objects/favorite.workspace-entity';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-participant.workspace-entity';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity';
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
import { OpportunityWorkspaceEntity } from 'src/modules/opportunity/standard-objects/opportunity.workspace-entity'; import { OpportunityWorkspaceEntity } from 'src/modules/opportunity/standard-objects/opportunity.workspace-entity';
import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity'; import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity';
import { ViewFieldWorkspaceEntity } from 'src/modules/view/standard-objects/view-field.workspace-entity'; import { ViewFieldWorkspaceEntity } from 'src/modules/view/standard-objects/view-field.workspace-entity';
@ -27,6 +22,11 @@ import { CalendarChannelEventAssociationWorkspaceEntity } from 'src/modules/cale
import { AuditLogWorkspaceEntity } from 'src/modules/timeline/standard-objects/audit-log.workspace-entity'; import { AuditLogWorkspaceEntity } from 'src/modules/timeline/standard-objects/audit-log.workspace-entity';
import { TimelineActivityWorkspaceEntity } from 'src/modules/timeline/standard-objects/timeline-activity.workspace-entity'; import { TimelineActivityWorkspaceEntity } from 'src/modules/timeline/standard-objects/timeline-activity.workspace-entity';
import { BehavioralEventWorkspaceEntity } from 'src/modules/timeline/standard-objects/behavioral-event.workspace-entity'; import { BehavioralEventWorkspaceEntity } from 'src/modules/timeline/standard-objects/behavioral-event.workspace-entity';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-participant.workspace-entity';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
export const standardObjectMetadataDefinitions = [ export const standardObjectMetadataDefinitions = [
ActivityTargetWorkspaceEntity, ActivityTargetWorkspaceEntity,

View File

@ -3,10 +3,10 @@ import { Module } from '@nestjs/common';
import { MatchParticipantJob } from 'src/modules/calendar-messaging-participant/jobs/match-participant.job'; import { MatchParticipantJob } from 'src/modules/calendar-messaging-participant/jobs/match-participant.job';
import { UnmatchParticipantJob } from 'src/modules/calendar-messaging-participant/jobs/unmatch-participant.job'; import { UnmatchParticipantJob } from 'src/modules/calendar-messaging-participant/jobs/unmatch-participant.job';
import { CalendarEventParticipantModule } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.module'; import { CalendarEventParticipantModule } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.module';
import { MessageParticipantModule } from 'src/modules/messaging/services/message-participant/message-participant.module'; import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
@Module({ @Module({
imports: [MessageParticipantModule, CalendarEventParticipantModule], imports: [CalendarEventParticipantModule, MessagingCommonModule],
providers: [ providers: [
{ {
provide: MatchParticipantJob.name, provide: MatchParticipantJob.name,

View File

@ -3,7 +3,7 @@ import { Injectable } from '@nestjs/common';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface'; import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { CalendarEventParticipantService } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.service'; import { CalendarEventParticipantService } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.service';
import { MessageParticipantService } from 'src/modules/messaging/services/message-participant/message-participant.service'; import { MessagingMessageParticipantService } from 'src/modules/messaging/common/services/messaging-message-participant.service';
export type MatchParticipantJobData = { export type MatchParticipantJobData = {
workspaceId: string; workspaceId: string;
@ -17,7 +17,7 @@ export class MatchParticipantJob
implements MessageQueueJob<MatchParticipantJobData> implements MessageQueueJob<MatchParticipantJobData>
{ {
constructor( constructor(
private readonly messageParticipantService: MessageParticipantService, private readonly messageParticipantService: MessagingMessageParticipantService,
private readonly calendarEventParticipantService: CalendarEventParticipantService, private readonly calendarEventParticipantService: CalendarEventParticipantService,
) {} ) {}

View File

@ -3,7 +3,7 @@ import { Injectable } from '@nestjs/common';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface'; import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { CalendarEventParticipantService } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.service'; import { CalendarEventParticipantService } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.service';
import { MessageParticipantService } from 'src/modules/messaging/services/message-participant/message-participant.service'; import { MessagingMessageParticipantService } from 'src/modules/messaging/common/services/messaging-message-participant.service';
export type UnmatchParticipantJobData = { export type UnmatchParticipantJobData = {
workspaceId: string; workspaceId: string;
@ -17,7 +17,7 @@ export class UnmatchParticipantJob
implements MessageQueueJob<UnmatchParticipantJobData> implements MessageQueueJob<UnmatchParticipantJobData>
{ {
constructor( constructor(
private readonly messageParticipantService: MessageParticipantService, private readonly messageParticipantService: MessagingMessageParticipantService,
private readonly calendarEventParticipantService: CalendarEventParticipantService, private readonly calendarEventParticipantService: CalendarEventParticipantService,
) {} ) {}

View File

@ -9,7 +9,7 @@ import {
CalendarCreateCompanyAndContactAfterSyncJobData, CalendarCreateCompanyAndContactAfterSyncJobData,
CalendarCreateCompanyAndContactAfterSyncJob, CalendarCreateCompanyAndContactAfterSyncJob,
} from 'src/modules/calendar/jobs/calendar-create-company-and-contact-after-sync.job'; } from 'src/modules/calendar/jobs/calendar-create-company-and-contact-after-sync.job';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
@Injectable() @Injectable()
export class CalendarChannelListener { export class CalendarChannelListener {

View File

@ -3,7 +3,7 @@ import { Injectable } from '@nestjs/common';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { CalendarEventRepository } from 'src/modules/calendar/repositories/calendar-event.repository'; import { CalendarEventRepository } from 'src/modules/calendar/repositories/calendar-event.repository';
import { CalendarEventWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-event.workspace-entity'; import { CalendarEventWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-event.workspace-entity';
import { deleteUsingPagination } from 'src/modules/messaging/services/thread-cleaner/utils/delete-using-pagination.util'; import { deleteUsingPagination } from 'src/modules/messaging/message-cleaner/utils/delete-using-pagination.util';
@Injectable() @Injectable()
export class CalendarEventCleanerService { export class CalendarEventCleanerService {

View File

@ -7,11 +7,11 @@ import { CreateContactModule } from 'src/modules/connected-account/auto-companie
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module'; import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity'; import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity';
import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity'; import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity';
import { MessageParticipantModule } from 'src/modules/messaging/services/message-participant/message-participant.module';
import { WorkspaceDataSourceModule } from 'src/engine/workspace-datasource/workspace-datasource.module'; import { WorkspaceDataSourceModule } from 'src/engine/workspace-datasource/workspace-datasource.module';
import { CalendarEventParticipantWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-event-participant.workspace-entity'; import { CalendarEventParticipantWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-event-participant.workspace-entity';
import { CalendarEventParticipantModule } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.module'; import { CalendarEventParticipantModule } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.module';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity'; import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
@Module({ @Module({
imports: [ imports: [
@ -22,7 +22,7 @@ import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-
WorkspaceMemberWorkspaceEntity, WorkspaceMemberWorkspaceEntity,
CalendarEventParticipantWorkspaceEntity, CalendarEventParticipantWorkspaceEntity,
]), ]),
MessageParticipantModule, MessagingCommonModule,
WorkspaceDataSourceModule, WorkspaceDataSourceModule,
CalendarEventParticipantModule, CalendarEventParticipantModule,
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'), TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),

View File

@ -5,11 +5,11 @@ import { ObjectRecordUpdateEvent } from 'src/engine/integrations/event-emitter/t
import { objectRecordChangedProperties } from 'src/engine/integrations/event-emitter/utils/object-record-changed-properties.util'; import { objectRecordChangedProperties } from 'src/engine/integrations/event-emitter/utils/object-record-changed-properties.util';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants'; import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service'; import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { import {
MessagingCreateCompanyAndContactAfterSyncJobData, MessagingCreateCompanyAndContactAfterSyncJobData,
MessagingCreateCompanyAndContactAfterSyncJob, MessagingCreateCompanyAndContactAfterSyncJob,
} from 'src/modules/messaging/jobs/messaging-create-company-and-contact-after-sync.job'; } from 'src/modules/messaging/message-participants-manager/jobs/messaging-create-company-and-contact-after-sync.job';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
@Injectable() @Injectable()
export class MessagingMessageChannelListener { export class MessagingMessageChannelListener {

View File

@ -1,7 +1,6 @@
import { Injectable } from '@nestjs/common'; import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { EntityManager, Repository } from 'typeorm'; import { EntityManager } from 'typeorm';
import compact from 'lodash.compact'; import compact from 'lodash.compact';
import { getDomainNameFromHandle } from 'src/modules/calendar-messaging-participant/utils/get-domain-name-from-handle.util'; import { getDomainNameFromHandle } from 'src/modules/calendar-messaging-participant/utils/get-domain-name-from-handle.util';
@ -16,11 +15,10 @@ import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/sta
import { getUniqueContactsAndHandles } from 'src/modules/connected-account/auto-companies-and-contacts-creation/utils/get-unique-contacts-and-handles.util'; import { getUniqueContactsAndHandles } from 'src/modules/connected-account/auto-companies-and-contacts-creation/utils/get-unique-contacts-and-handles.util';
import { Contacts } from 'src/modules/connected-account/auto-companies-and-contacts-creation/types/contact.type'; import { Contacts } from 'src/modules/connected-account/auto-companies-and-contacts-creation/types/contact.type';
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service'; import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
import { MessageParticipantService } from 'src/modules/messaging/services/message-participant/message-participant.service';
import { CalendarEventParticipantService } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.service'; import { CalendarEventParticipantService } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.service';
import { filterOutContactsFromCompanyOrWorkspace } from 'src/modules/connected-account/auto-companies-and-contacts-creation/utils/filter-out-contacts-from-company-or-workspace.util'; import { filterOutContactsFromCompanyOrWorkspace } from 'src/modules/connected-account/auto-companies-and-contacts-creation/utils/filter-out-contacts-from-company-or-workspace.util';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { MessagingMessageParticipantService } from 'src/modules/messaging/common/services/messaging-message-participant.service';
@Injectable() @Injectable()
export class CreateCompanyAndContactService { export class CreateCompanyAndContactService {
@ -32,10 +30,8 @@ export class CreateCompanyAndContactService {
@InjectObjectMetadataRepository(WorkspaceMemberWorkspaceEntity) @InjectObjectMetadataRepository(WorkspaceMemberWorkspaceEntity)
private readonly workspaceMemberRepository: WorkspaceMemberRepository, private readonly workspaceMemberRepository: WorkspaceMemberRepository,
private readonly workspaceDataSourceService: WorkspaceDataSourceService, private readonly workspaceDataSourceService: WorkspaceDataSourceService,
private readonly messageParticipantService: MessageParticipantService, private readonly messageParticipantService: MessagingMessageParticipantService,
private readonly calendarEventParticipantService: CalendarEventParticipantService, private readonly calendarEventParticipantService: CalendarEventParticipantService,
@InjectRepository(FeatureFlagEntity, 'core')
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
) {} ) {}
async createCompaniesAndPeople( async createCompaniesAndPeople(

View File

@ -3,14 +3,14 @@ import { Module } from '@nestjs/common';
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module'; import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
import { GoogleAPIRefreshAccessTokenService } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.service'; import { GoogleAPIRefreshAccessTokenService } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.service';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { GmailErrorHandlingModule } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.module'; import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
@Module({ @Module({
imports: [ imports: [
ObjectMetadataRepositoryModule.forFeature([ ObjectMetadataRepositoryModule.forFeature([
ConnectedAccountWorkspaceEntity, ConnectedAccountWorkspaceEntity,
]), ]),
GmailErrorHandlingModule, MessagingCommonModule,
], ],
providers: [GoogleAPIRefreshAccessTokenService], providers: [GoogleAPIRefreshAccessTokenService],
exports: [GoogleAPIRefreshAccessTokenService], exports: [GoogleAPIRefreshAccessTokenService],

View File

@ -6,9 +6,10 @@ import { EnvironmentService } from 'src/engine/integrations/environment/environm
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository'; import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository'; import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { GmailErrorHandlingService } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service'; import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
@Injectable() @Injectable()
export class GoogleAPIRefreshAccessTokenService { export class GoogleAPIRefreshAccessTokenService {
@ -18,7 +19,8 @@ export class GoogleAPIRefreshAccessTokenService {
private readonly connectedAccountRepository: ConnectedAccountRepository, private readonly connectedAccountRepository: ConnectedAccountRepository,
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity) @InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
private readonly messageChannelRepository: MessageChannelRepository, private readonly messageChannelRepository: MessageChannelRepository,
private readonly gmailErrorHandlingService: GmailErrorHandlingService, private readonly messagingTelemetryService: MessagingTelemetryService,
private readonly messagingChannelSyncStatusService: MessagingChannelSyncStatusService,
) {} ) {}
async refreshAndSaveAccessToken( async refreshAndSaveAccessToken(
@ -65,13 +67,21 @@ export class GoogleAPIRefreshAccessTokenService {
); );
} }
await this.gmailErrorHandlingService.handleGmailError( await this.messagingTelemetryService.track({
{ eventName: `refresh-token.error.insufficient_permissions`,
code: error.code, workspaceId,
reason: error.response.data.error, connectedAccountId: messageChannel.connectedAccountId,
}, messageChannelId: messageChannel.id,
'messages-import', message: `${error.code}: ${error.reason}`,
messageChannel, });
await this.messagingChannelSyncStatusService.markAsFailedInsufficientPermissionsAndFlushMessagesToImport(
messageChannel.id,
workspaceId,
);
await this.connectedAccountRepository.updateAuthFailedAt(
messageChannel.connectedAccountId,
workspaceId, workspaceId,
); );
} }

View File

@ -8,7 +8,6 @@ import {
import { CONNECTED_ACCOUNT_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids'; import { CONNECTED_ACCOUNT_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids'; import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
import { CalendarChannelWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-channel.workspace-entity'; import { CalendarChannelWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-channel.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity'; import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity';
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity'; import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator'; import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator';
@ -17,6 +16,7 @@ import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/work
import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field.decorator'; import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field.decorator';
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator'; import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator'; import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
export enum ConnectedAccountProvider { export enum ConnectedAccountProvider {
GOOGLE = 'google', GOOGLE = 'google',

View File

@ -5,11 +5,11 @@ import { MessageQueueJob } from 'src/engine/integrations/message-queue/interface
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { BlocklistRepository } from 'src/modules/connected-account/repositories/blocklist.repository'; import { BlocklistRepository } from 'src/modules/connected-account/repositories/blocklist.repository';
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity'; import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository'; import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository'; import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { ThreadCleanerService } from 'src/modules/messaging/services/thread-cleaner/thread-cleaner.service'; import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity'; import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; import { MessagingMessageCleanerService } from 'src/modules/messaging/message-cleaner/services/messaging-message-cleaner.service';
export type BlocklistItemDeleteMessagesJobData = { export type BlocklistItemDeleteMessagesJobData = {
workspaceId: string; workspaceId: string;
@ -31,7 +31,7 @@ export class BlocklistItemDeleteMessagesJob
private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository, private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository,
@InjectObjectMetadataRepository(BlocklistWorkspaceEntity) @InjectObjectMetadataRepository(BlocklistWorkspaceEntity)
private readonly blocklistRepository: BlocklistRepository, private readonly blocklistRepository: BlocklistRepository,
private readonly threadCleanerService: ThreadCleanerService, private readonly threadCleanerService: MessagingMessageCleanerService,
) {} ) {}
async handle(data: BlocklistItemDeleteMessagesJobData): Promise<void> { async handle(data: BlocklistItemDeleteMessagesJobData): Promise<void> {

View File

@ -5,7 +5,6 @@ import { MessageQueueJob } from 'src/engine/integrations/message-queue/interface
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository'; import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { GmailFullMessageListFetchService } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch.service';
export type BlocklistReimportMessagesJobData = { export type BlocklistReimportMessagesJobData = {
workspaceId: string; workspaceId: string;
@ -22,7 +21,6 @@ export class BlocklistReimportMessagesJob
constructor( constructor(
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity) @InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
private readonly connectedAccountRepository: ConnectedAccountRepository, private readonly connectedAccountRepository: ConnectedAccountRepository,
private readonly gmailFullMessageListFetchService: GmailFullMessageListFetchService,
) {} ) {}
async handle(data: BlocklistReimportMessagesJobData): Promise<void> { async handle(data: BlocklistReimportMessagesJobData): Promise<void> {
@ -46,11 +44,12 @@ export class BlocklistReimportMessagesJob
return; return;
} }
await this.gmailFullMessageListFetchService.fetchConnectedAccountThreads( // TODO: reimplement that
workspaceId, // await this.gmailMessageListFetchJob.fetchConnectedAccountThreads(
connectedAccount[0].id, // workspaceId,
[handle], // connectedAccount[0].id,
); // [handle],
// );
this.logger.log( this.logger.log(
`Reimporting messages from ${handle} in workspace ${workspaceId} for workspace member ${workspaceMemberId} done`, `Reimporting messages from ${handle} in workspace ${workspaceId} for workspace member ${workspaceMemberId} done`,

View File

@ -9,11 +9,11 @@ import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard
import { import {
BlocklistReimportMessagesJob, BlocklistReimportMessagesJob,
BlocklistReimportMessagesJobData, BlocklistReimportMessagesJobData,
} from 'src/modules/messaging/jobs/blocklist-reimport-messages.job'; } from 'src/modules/messaging/blocklist-manager/jobs/messaging-blocklist-reimport-messages.job';
import { import {
BlocklistItemDeleteMessagesJobData, BlocklistItemDeleteMessagesJobData,
BlocklistItemDeleteMessagesJob, BlocklistItemDeleteMessagesJob,
} from 'src/modules/messaging/jobs/blocklist-item-delete-messages.job'; } from 'src/modules/messaging/blocklist-manager/jobs/messaging-blocklist-item-delete-messages.job';
import { ObjectRecordUpdateEvent } from 'src/engine/integrations/event-emitter/types/object-record-update.event'; import { ObjectRecordUpdateEvent } from 'src/engine/integrations/event-emitter/types/object-record-update.event';
@Injectable() @Injectable()

View File

@ -0,0 +1,8 @@
import { Module } from '@nestjs/common';
@Module({
imports: [],
providers: [],
exports: [],
})
export class MessagingBlocklistManagerModule {}

View File

@ -0,0 +1,60 @@
import { HttpModule } from '@nestjs/axios';
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { AnalyticsModule } from 'src/engine/core-modules/analytics/analytics.module';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
import { WorkspaceDataSourceModule } from 'src/engine/workspace-datasource/workspace-datasource.module';
import { AddPersonIdAndWorkspaceMemberIdModule } from 'src/modules/calendar-messaging-participant/services/add-person-id-and-workspace-member-id/add-person-id-and-workspace-member-id.module';
import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
import { MessagingErrorHandlingService } from 'src/modules/messaging/common/services/messaging-error-handling.service';
import { MessagingFetchByBatchesService } from 'src/modules/messaging/common/services/messaging-fetch-by-batch.service';
import { MessagingMessageParticipantService } from 'src/modules/messaging/common/services/messaging-message-participant.service';
import { MessagingMessageThreadService } from 'src/modules/messaging/common/services/messaging-message-thread.service';
import { MessagingMessageService } from 'src/modules/messaging/common/services/messaging-message.service';
import { MessagingSaveMessagesAndEnqueueContactCreationService } from 'src/modules/messaging/common/services/messaging-save-messages-and-enqueue-contact-creation.service';
import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-participant.workspace-entity';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity';
@Module({
imports: [
HttpModule.register({
baseURL: 'https://www.googleapis.com/batch/gmail/v1',
}),
AnalyticsModule,
WorkspaceDataSourceModule,
ObjectMetadataRepositoryModule.forFeature([
PersonWorkspaceEntity,
MessageParticipantWorkspaceEntity,
MessageWorkspaceEntity,
MessageThreadWorkspaceEntity,
]),
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
AddPersonIdAndWorkspaceMemberIdModule,
],
providers: [
MessagingMessageService,
MessagingMessageThreadService,
MessagingSaveMessagesAndEnqueueContactCreationService,
MessagingErrorHandlingService,
MessagingTelemetryService,
MessagingChannelSyncStatusService,
MessagingMessageParticipantService,
MessagingFetchByBatchesService,
],
exports: [
MessagingMessageService,
MessagingMessageThreadService,
MessagingSaveMessagesAndEnqueueContactCreationService,
MessagingErrorHandlingService,
MessagingTelemetryService,
MessagingChannelSyncStatusService,
MessagingMessageParticipantService,
MessagingFetchByBatchesService,
],
})
export class MessagingCommonModule {}

View File

@ -10,15 +10,15 @@ import groupBy from 'lodash.groupby';
import { WorkspacePreQueryHook } from 'src/engine/api/graphql/workspace-query-runner/workspace-pre-query-hook/interfaces/workspace-pre-query-hook.interface'; import { WorkspacePreQueryHook } from 'src/engine/api/graphql/workspace-query-runner/workspace-pre-query-hook/interfaces/workspace-pre-query-hook.interface';
import { FindManyResolverArgs } from 'src/engine/api/graphql/workspace-resolver-builder/interfaces/workspace-resolvers-builder.interface'; import { FindManyResolverArgs } from 'src/engine/api/graphql/workspace-resolver-builder/interfaces/workspace-resolvers-builder.interface';
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository'; import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
import { WorkspaceMemberRepository } from 'src/modules/workspace-member/repositories/workspace-member.repository'; import { WorkspaceMemberRepository } from 'src/modules/workspace-member/repositories/workspace-member.repository';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity'; import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity';
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
@Injectable() @Injectable()
export class MessageFindManyPreQueryHook implements WorkspacePreQueryHook { export class MessageFindManyPreQueryHook implements WorkspacePreQueryHook {

View File

@ -1,12 +1,12 @@
import { Module } from '@nestjs/common'; import { Module } from '@nestjs/common';
import { MessageFindManyPreQueryHook } from 'src/modules/messaging/query-hooks/message/message-find-many.pre-query.hook';
import { MessageFindOnePreQueryHook } from 'src/modules/messaging/query-hooks/message/message-find-one.pre-query-hook';
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module'; import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity'; import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; import { MessageFindManyPreQueryHook } from 'src/modules/messaging/common/query-hooks/message/message-find-many.pre-query.hook';
import { MessageFindOnePreQueryHook } from 'src/modules/messaging/common/query-hooks/message/message-find-one.pre-query-hook';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
@Module({ @Module({
imports: [ imports: [

View File

@ -3,8 +3,8 @@ import { Injectable } from '@nestjs/common';
import { EntityManager } from 'typeorm'; import { EntityManager } from 'typeorm';
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service'; import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
@Injectable() @Injectable()
export class MessageChannelMessageAssociationRepository { export class MessageChannelMessageAssociationRepository {

View File

@ -3,12 +3,12 @@ import { Injectable } from '@nestjs/common';
import { EntityManager } from 'typeorm'; import { EntityManager } from 'typeorm';
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service'; import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { import {
MessageChannelWorkspaceEntity, MessageChannelWorkspaceEntity,
MessageChannelSyncStatus, MessageChannelSyncStatus,
MessageChannelSyncSubStatus, MessageChannelSyncSubStatus,
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
@Injectable() @Injectable()
export class MessageChannelRepository { export class MessageChannelRepository {

View File

@ -3,9 +3,9 @@ import { Injectable } from '@nestjs/common';
import { EntityManager } from 'typeorm'; import { EntityManager } from 'typeorm';
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service'; import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-participant.workspace-entity';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { ParticipantWithId } from 'src/modules/messaging/types/gmail-message'; import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-participant.workspace-entity';
import { ParticipantWithId } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
@Injectable() @Injectable()
export class MessageParticipantRepository { export class MessageParticipantRepository {

View File

@ -3,8 +3,8 @@ import { Injectable } from '@nestjs/common';
import { EntityManager } from 'typeorm'; import { EntityManager } from 'typeorm';
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service'; import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
@Injectable() @Injectable()
export class MessageRepository { export class MessageRepository {

View File

@ -4,15 +4,15 @@ import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache
import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator'; import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator';
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum'; import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository'; import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { import {
MessageChannelWorkspaceEntity, MessageChannelWorkspaceEntity,
MessageChannelSyncSubStatus, MessageChannelSyncSubStatus,
MessageChannelSyncStatus, MessageChannelSyncStatus,
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
@Injectable() @Injectable()
export class MessageChannelSyncStatusService { export class MessagingChannelSyncStatusService {
constructor( constructor(
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity) @InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
private readonly messageChannelRepository: MessageChannelRepository, private readonly messageChannelRepository: MessageChannelRepository,

View File

@ -6,9 +6,9 @@ import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repos
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository'; import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelSyncStatusService } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.service'; import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
import { MessagingTelemetryService } from 'src/modules/messaging/services/telemetry/messaging-telemetry.service'; import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
type SyncStep = type SyncStep =
| 'partial-message-list-fetch' | 'partial-message-list-fetch'
@ -21,11 +21,11 @@ export type GmailError = {
}; };
@Injectable() @Injectable()
export class GmailErrorHandlingService { export class MessagingErrorHandlingService {
constructor( constructor(
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity) @InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
private readonly connectedAccountRepository: ConnectedAccountRepository, private readonly connectedAccountRepository: ConnectedAccountRepository,
private readonly messageChannelSyncStatusService: MessageChannelSyncStatusService, private readonly messagingChannelSyncStatusService: MessagingChannelSyncStatusService,
private readonly messagingTelemetryService: MessagingTelemetryService, private readonly messagingTelemetryService: MessagingTelemetryService,
) {} ) {}
@ -92,7 +92,7 @@ export class GmailErrorHandlingService {
break; break;
default: default:
await this.messageChannelSyncStatusService.markAsFailedUnknownAndFlushMessagesToImport( await this.messagingChannelSyncStatusService.markAsFailedUnknownAndFlushMessagesToImport(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
@ -116,21 +116,21 @@ export class GmailErrorHandlingService {
switch (syncStep) { switch (syncStep) {
case 'full-message-list-fetch': case 'full-message-list-fetch':
await this.messageChannelSyncStatusService.scheduleFullMessageListFetch( await this.messagingChannelSyncStatusService.scheduleFullMessageListFetch(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
break; break;
case 'partial-message-list-fetch': case 'partial-message-list-fetch':
await this.messageChannelSyncStatusService.schedulePartialMessageListFetch( await this.messagingChannelSyncStatusService.schedulePartialMessageListFetch(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
break; break;
case 'messages-import': case 'messages-import':
await this.messageChannelSyncStatusService.scheduleMessagesImport( await this.messagingChannelSyncStatusService.scheduleMessagesImport(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
@ -155,7 +155,7 @@ export class GmailErrorHandlingService {
message: `${error.code}: ${error.reason}`, message: `${error.code}: ${error.reason}`,
}); });
await this.messageChannelSyncStatusService.markAsFailedInsufficientPermissionsAndFlushMessagesToImport( await this.messagingChannelSyncStatusService.markAsFailedInsufficientPermissionsAndFlushMessagesToImport(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
@ -184,7 +184,7 @@ export class GmailErrorHandlingService {
message: `404: ${error.reason}`, message: `404: ${error.reason}`,
}); });
await this.messageChannelSyncStatusService.resetAndScheduleFullMessageListFetch( await this.messagingChannelSyncStatusService.resetAndScheduleFullMessageListFetch(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );

View File

@ -3,11 +3,11 @@ import { Injectable } from '@nestjs/common';
import { AxiosResponse } from 'axios'; import { AxiosResponse } from 'axios';
import { BatchQueries } from 'src/modules/messaging/types/batch-queries'; import { GmailMessageParsedResponse } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message-parsed-response';
import { GmailMessageParsedResponse } from 'src/modules/messaging/types/gmail-message-parsed-response'; import { BatchQueries } from 'src/modules/messaging/message-import-manager/types/batch-queries';
@Injectable() @Injectable()
export class FetchByBatchesService { export class MessagingFetchByBatchesService {
constructor(private readonly httpService: HttpService) {} constructor(private readonly httpService: HttpService) {}
async fetchAllByBatches( async fetchAllByBatches(

View File

@ -3,18 +3,20 @@ import { Injectable } from '@nestjs/common';
import { EntityManager } from 'typeorm'; import { EntityManager } from 'typeorm';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { ParticipantWithMessageId } from 'src/modules/messaging/types/gmail-message';
import { PersonRepository } from 'src/modules/person/repositories/person.repository'; import { PersonRepository } from 'src/modules/person/repositories/person.repository';
import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity'; import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity';
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service'; import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
import { getFlattenedValuesAndValuesStringForBatchRawQuery } from 'src/modules/calendar/utils/get-flattened-values-and-values-string-for-batch-raw-query.util'; import { getFlattenedValuesAndValuesStringForBatchRawQuery } from 'src/modules/calendar/utils/get-flattened-values-and-values-string-for-batch-raw-query.util';
import { MessageParticipantRepository } from 'src/modules/messaging/repositories/message-participant.repository';
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-participant.workspace-entity';
import { AddPersonIdAndWorkspaceMemberIdService } from 'src/modules/calendar-messaging-participant/services/add-person-id-and-workspace-member-id/add-person-id-and-workspace-member-id.service'; import { AddPersonIdAndWorkspaceMemberIdService } from 'src/modules/calendar-messaging-participant/services/add-person-id-and-workspace-member-id/add-person-id-and-workspace-member-id.service';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { MessageParticipantRepository } from 'src/modules/messaging/common/repositories/message-participant.repository';
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-participant.workspace-entity';
import { ParticipantWithMessageId } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
// Todo: this is not the right place for this file. The code needs to be refactored in term of business modules with a precise scope.
// Putting it here to avoid circular dependencies for now.
@Injectable() @Injectable()
export class MessageParticipantService { export class MessagingMessageParticipantService {
constructor( constructor(
private readonly workspaceDataSourceService: WorkspaceDataSourceService, private readonly workspaceDataSourceService: WorkspaceDataSourceService,
@InjectObjectMetadataRepository(MessageParticipantWorkspaceEntity) @InjectObjectMetadataRepository(MessageParticipantWorkspaceEntity)

View File

@ -4,15 +4,15 @@ import { EntityManager } from 'typeorm';
import { v4 } from 'uuid'; import { v4 } from 'uuid';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository'; import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
import { MessageRepository } from 'src/modules/messaging/repositories/message.repository'; import { MessageThreadRepository } from 'src/modules/messaging/common/repositories/message-thread.repository';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity'; import { MessageRepository } from 'src/modules/messaging/common/repositories/message.repository';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity'; import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageThreadRepository } from 'src/modules/messaging/repositories/message-thread.repository'; import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity'; import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
@Injectable() @Injectable()
export class MessageThreadService { export class MessagingMessageThreadService {
constructor( constructor(
@InjectObjectMetadataRepository( @InjectObjectMetadataRepository(
MessageChannelMessageAssociationWorkspaceEntity, MessageChannelMessageAssociationWorkspaceEntity,

View File

@ -4,23 +4,23 @@ import { DataSource, EntityManager } from 'typeorm';
import { v4 } from 'uuid'; import { v4 } from 'uuid';
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service'; import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { GmailMessage } from 'src/modules/messaging/types/gmail-message';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository'; import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
import { MessageRepository } from 'src/modules/messaging/repositories/message.repository'; import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity'; import { MessageThreadRepository } from 'src/modules/messaging/common/repositories/message-thread.repository';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity'; import { MessageRepository } from 'src/modules/messaging/common/repositories/message.repository';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository'; import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageThreadService } from 'src/modules/messaging/services/message-thread/message-thread.service'; import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity'; import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
import { MessageThreadRepository } from 'src/modules/messaging/repositories/message-thread.repository'; import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
import { GmailMessage } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
import { MessagingMessageThreadService } from 'src/modules/messaging/common/services/messaging-message-thread.service';
@Injectable() @Injectable()
export class MessageService { export class MessagingMessageService {
private readonly logger = new Logger(MessageService.name); private readonly logger = new Logger(MessagingMessageService.name);
constructor( constructor(
private readonly workspaceDataSourceService: WorkspaceDataSourceService, private readonly workspaceDataSourceService: WorkspaceDataSourceService,
@ -34,7 +34,7 @@ export class MessageService {
private readonly messageChannelRepository: MessageChannelRepository, private readonly messageChannelRepository: MessageChannelRepository,
@InjectObjectMetadataRepository(MessageThreadWorkspaceEntity) @InjectObjectMetadataRepository(MessageThreadWorkspaceEntity)
private readonly messageThreadRepository: MessageThreadRepository, private readonly messageThreadRepository: MessageThreadRepository,
private readonly messageThreadService: MessageThreadService, private readonly messageThreadService: MessagingMessageThreadService,
) {} ) {}
public async saveMessagesWithinTransaction( public async saveMessagesWithinTransaction(

View File

@ -8,13 +8,6 @@ import { MessageQueueService } from 'src/engine/integrations/message-queue/servi
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service'; import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageParticipantService } from 'src/modules/messaging/services/message-participant/message-participant.service';
import { MessageService } from 'src/modules/messaging/services/message/message.service';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import {
GmailMessage,
ParticipantWithMessageId,
} from 'src/modules/messaging/types/gmail-message';
import { import {
CreateCompanyAndContactJobData, CreateCompanyAndContactJobData,
CreateCompanyAndContactJob, CreateCompanyAndContactJob,
@ -23,15 +16,22 @@ import {
FeatureFlagEntity, FeatureFlagEntity,
FeatureFlagKeys, FeatureFlagKeys,
} from 'src/engine/core-modules/feature-flag/feature-flag.entity'; } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import {
GmailMessage,
ParticipantWithMessageId,
} from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
import { MessagingMessageService } from 'src/modules/messaging/common/services/messaging-message.service';
import { MessagingMessageParticipantService } from 'src/modules/messaging/common/services/messaging-message-participant.service';
@Injectable() @Injectable()
export class SaveMessagesAndEnqueueContactCreationService { export class MessagingSaveMessagesAndEnqueueContactCreationService {
constructor( constructor(
private readonly workspaceDataSourceService: WorkspaceDataSourceService, private readonly workspaceDataSourceService: WorkspaceDataSourceService,
@Inject(MessageQueue.messagingQueue) @Inject(MessageQueue.messagingQueue)
private readonly messageQueueService: MessageQueueService, private readonly messageQueueService: MessageQueueService,
private readonly messageService: MessageService, private readonly messageService: MessagingMessageService,
private readonly messageParticipantService: MessageParticipantService, private readonly messageParticipantService: MessagingMessageParticipantService,
@InjectRepository(FeatureFlagEntity, 'core') @InjectRepository(FeatureFlagEntity, 'core')
private readonly featureFlagRepository: Repository<FeatureFlagEntity>, private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
) {} ) {}

View File

@ -3,9 +3,6 @@ import { Relation } from 'src/engine/workspace-manager/workspace-sync-metadata/i
import { FieldMetadataType } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity'; import { FieldMetadataType } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
import { MESSAGE_CHANNEL_MESSAGE_ASSOCIATION_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids'; import { MESSAGE_CHANNEL_MESSAGE_ASSOCIATION_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids'; import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity';
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity'; import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator'; import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator';
import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator'; import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator';
@ -14,6 +11,9 @@ import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator'; import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator'; import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
import { RelationMetadataType } from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity'; import { RelationMetadataType } from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
@WorkspaceEntity({ @WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.messageChannelMessageAssociation, standardId: STANDARD_OBJECT_IDS.messageChannelMessageAssociation,

View File

@ -8,7 +8,6 @@ import {
import { MESSAGE_CHANNEL_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids'; import { MESSAGE_CHANNEL_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids'; import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity'; import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator'; import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator';
import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator'; import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator';
@ -16,6 +15,7 @@ import { WorkspaceIsSystem } from 'src/engine/twenty-orm/decorators/workspace-is
import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field.decorator'; import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field.decorator';
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator'; import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator'; import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
export enum MessageChannelSyncStatus { export enum MessageChannelSyncStatus {
// TO BE DEPRECATED // TO BE DEPRECATED

View File

@ -3,7 +3,6 @@ import { Relation } from 'src/engine/workspace-manager/workspace-sync-metadata/i
import { FieldMetadataType } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity'; import { FieldMetadataType } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
import { MESSAGE_PARTICIPANT_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids'; import { MESSAGE_PARTICIPANT_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids'; import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity'; import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity';
import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity'; import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity';
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity'; import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
@ -14,6 +13,7 @@ import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator'; import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
import { RelationMetadataType } from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity'; import { RelationMetadataType } from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity';
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator'; import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
@WorkspaceEntity({ @WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.messageParticipant, standardId: STANDARD_OBJECT_IDS.messageParticipant,

View File

@ -6,14 +6,14 @@ import {
} from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity'; } from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity';
import { MESSAGE_THREAD_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids'; import { MESSAGE_THREAD_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids'; import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity'; import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator'; import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator';
import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator'; import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator';
import { WorkspaceIsSystem } from 'src/engine/twenty-orm/decorators/workspace-is-system.decorator'; import { WorkspaceIsSystem } from 'src/engine/twenty-orm/decorators/workspace-is-system.decorator';
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator'; import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator'; import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
@WorkspaceEntity({ @WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.messageThread, standardId: STANDARD_OBJECT_IDS.messageThread,

View File

@ -7,9 +7,6 @@ import {
} from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity'; } from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity';
import { MESSAGE_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids'; import { MESSAGE_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids'; import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-participant.workspace-entity';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity';
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity'; import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator'; import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator';
import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator'; import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator';
@ -17,6 +14,9 @@ import { WorkspaceIsSystem } from 'src/engine/twenty-orm/decorators/workspace-is
import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field.decorator'; import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field.decorator';
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator'; import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator'; import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-participant.workspace-entity';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
@WorkspaceEntity({ @WorkspaceEntity({
standardId: STANDARD_OBJECT_IDS.message, standardId: STANDARD_OBJECT_IDS.message,

View File

@ -1 +0,0 @@
export const GMAIL_EXCLUDED_CATEGORIES = ['promotions', 'social', 'forums'];

View File

@ -1 +0,0 @@
export const GMAIL_ONGOING_SYNC_TIMEOUT = 1000 * 60 * 60; // 1 hour

View File

@ -1 +0,0 @@
export const GMAIL_USERS_HISTORY_MAX_RESULT = 500;

View File

@ -1 +0,0 @@
export const GMAIL_USERS_MESSAGES_GET_BATCH_SIZE = 20;

View File

@ -1 +0,0 @@
export const GMAIL_USERS_MESSAGES_LIST_MAX_RESULT = 500;

View File

@ -1,15 +0,0 @@
import { Module } from '@nestjs/common';
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { GmailMessagesImportCronCommand } from 'src/modules/messaging/crons/commands/gmail-messages-import.cron.command';
import { GmailMessageListFetchCronCommand } from 'src/modules/messaging/crons/commands/gmail-message-list-fetch.cron.command';
@Module({
imports: [
ObjectMetadataRepositoryModule.forFeature([
ConnectedAccountWorkspaceEntity,
]),
],
providers: [GmailMessageListFetchCronCommand, GmailMessagesImportCronCommand],
})
export class MessagingCronCommandsModule {}

View File

@ -1,117 +0,0 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository, In } from 'typeorm';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { GmailMessagesImportService } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import.service';
import { EnvironmentService } from 'src/engine/integrations/environment/environment.service';
import { GmailMessagesImportV2Service } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import-v2.service';
import {
FeatureFlagEntity,
FeatureFlagKeys,
} from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
import { MessagingTelemetryService } from 'src/modules/messaging/services/telemetry/messaging-telemetry.service';
@Injectable()
export class GmailMessagesImportCronJob implements MessageQueueJob<undefined> {
private readonly logger = new Logger(GmailMessagesImportCronJob.name);
constructor(
@InjectRepository(Workspace, 'core')
private readonly workspaceRepository: Repository<Workspace>,
@InjectRepository(DataSourceEntity, 'metadata')
private readonly dataSourceRepository: Repository<DataSourceEntity>,
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
private readonly messageChannelRepository: MessageChannelRepository,
private readonly gmailFetchMessageContentFromCacheService: GmailMessagesImportService,
private readonly gmailFetchMessageContentFromCacheV2Service: GmailMessagesImportV2Service,
@InjectRepository(FeatureFlagEntity, 'core')
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
private readonly environmentService: EnvironmentService,
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
private readonly connectedAccountRepository: ConnectedAccountRepository,
private readonly messagingTelemetryService: MessagingTelemetryService,
) {}
async handle(): Promise<void> {
const workspaceIds = (
await this.workspaceRepository.find({
where: this.environmentService.get('IS_BILLING_ENABLED')
? {
subscriptionStatus: In(['active', 'trialing', 'past_due']),
}
: {},
select: ['id'],
})
).map((workspace) => workspace.id);
const dataSources = await this.dataSourceRepository.find({
where: {
workspaceId: In(workspaceIds),
},
});
const workspaceIdsWithDataSources = new Set(
dataSources.map((dataSource) => dataSource.workspaceId),
);
for (const workspaceId of workspaceIdsWithDataSources) {
await this.fetchWorkspaceMessages(workspaceId);
}
}
private async fetchWorkspaceMessages(workspaceId: string): Promise<void> {
const messageChannels =
await this.messageChannelRepository.getAll(workspaceId);
const isGmailSyncV2EnabledFeatureFlag =
await this.featureFlagRepository.findOneBy({
workspaceId: workspaceId,
key: FeatureFlagKeys.IsGmailSyncV2Enabled,
value: true,
});
const isGmailSyncV2Enabled = isGmailSyncV2EnabledFeatureFlag?.value;
for (const messageChannel of messageChannels) {
if (!messageChannel?.isSyncEnabled) {
continue;
}
if (isGmailSyncV2Enabled) {
await this.messagingTelemetryService.track({
eventName: 'messages_import.triggered',
workspaceId,
connectedAccountId: messageChannel.connectedAccountId,
messageChannelId: messageChannel.id,
});
const connectedAccount =
await this.connectedAccountRepository.getConnectedAccountOrThrow(
workspaceId,
messageChannel.connectedAccountId,
);
await this.gmailFetchMessageContentFromCacheV2Service.processMessageBatchImport(
messageChannel,
connectedAccount,
workspaceId,
);
} else {
await this.gmailFetchMessageContentFromCacheService.fetchMessageContentFromCache(
workspaceId,
messageChannel.connectedAccountId,
);
}
}
}
}

View File

@ -1,33 +0,0 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity';
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
import { GmailMessagesImportCronJob } from 'src/modules/messaging/crons/jobs/gmail-messages-import.cron.job';
import { GmailMessageListFetchCronJob } from 'src/modules/messaging/crons/jobs/gmail-message-list-fetch.cron.job';
import { GmailMessagesImportModule } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import.module';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { MessagingTelemetryModule } from 'src/modules/messaging/services/telemetry/messaging-telemetry.module';
@Module({
imports: [
TypeOrmModule.forFeature([Workspace, FeatureFlagEntity], 'core'),
TypeOrmModule.forFeature([DataSourceEntity], 'metadata'),
ObjectMetadataRepositoryModule.forFeature([MessageChannelWorkspaceEntity]),
GmailMessagesImportModule,
MessagingTelemetryModule,
],
providers: [
{
provide: GmailMessagesImportCronJob.name,
useClass: GmailMessagesImportCronJob,
},
{
provide: GmailMessageListFetchCronJob.name,
useClass: GmailMessageListFetchCronJob,
},
],
})
export class MessagingCronJobModule {}

View File

@ -1,126 +0,0 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import {
FeatureFlagEntity,
FeatureFlagKeys,
} from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { GoogleAPIRefreshAccessTokenService } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.service';
import { GmailFullMessageListFetchService } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch.service';
import { GmailFullMessageListFetchV2Service } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch-v2.service';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { MessagingTelemetryService } from 'src/modules/messaging/services/telemetry/messaging-telemetry.service';
export type GmailFullMessageListFetchJobData = {
workspaceId: string;
connectedAccountId: string;
};
@Injectable()
export class GmailFullMessageListFetchJob
implements MessageQueueJob<GmailFullMessageListFetchJobData>
{
private readonly logger = new Logger(GmailFullMessageListFetchJob.name);
constructor(
private readonly googleAPIsRefreshAccessTokenService: GoogleAPIRefreshAccessTokenService,
private readonly gmailFullMessageListFetchService: GmailFullMessageListFetchService,
@InjectRepository(FeatureFlagEntity, 'core')
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
private readonly gmailFullMessageListFetchV2Service: GmailFullMessageListFetchV2Service,
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
private readonly connectedAccountRepository: ConnectedAccountRepository,
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
private readonly messageChannelRepository: MessageChannelRepository,
private readonly messagingTelemetryService: MessagingTelemetryService,
) {}
async handle(data: GmailFullMessageListFetchJobData): Promise<void> {
const { workspaceId, connectedAccountId } = data;
this.logger.log(
`gmail full-sync for workspace ${workspaceId} and account ${connectedAccountId}`,
);
try {
await this.googleAPIsRefreshAccessTokenService.refreshAndSaveAccessToken(
workspaceId,
connectedAccountId,
);
} catch (e) {
this.logger.error(
`Error refreshing access token for connected account ${connectedAccountId} in workspace ${workspaceId}`,
e,
);
return;
}
const isGmailSyncV2EnabledFeatureFlag =
await this.featureFlagRepository.findOneBy({
workspaceId: workspaceId,
key: FeatureFlagKeys.IsGmailSyncV2Enabled,
value: true,
});
const isGmailSyncV2Enabled = isGmailSyncV2EnabledFeatureFlag?.value;
if (isGmailSyncV2Enabled) {
// Todo delete this code block after migration
const connectedAccount = await this.connectedAccountRepository.getById(
connectedAccountId,
workspaceId,
);
if (!connectedAccount) {
throw new Error(
`Connected account ${connectedAccountId} not found in workspace ${workspaceId}`,
);
}
const messageChannel =
await this.messageChannelRepository.getFirstByConnectedAccountId(
connectedAccountId,
workspaceId,
);
if (!messageChannel) {
throw new Error(
`No message channel found for connected account ${connectedAccountId} in workspace ${workspaceId}`,
);
}
await this.messagingTelemetryService.track({
eventName: 'full_message_list_fetch.started',
workspaceId,
connectedAccountId,
});
await this.gmailFullMessageListFetchV2Service.processMessageListFetch(
messageChannel,
connectedAccount,
workspaceId,
);
await this.messagingTelemetryService.track({
eventName: 'full_message_list_fetch.completed',
workspaceId,
connectedAccountId,
messageChannelId: messageChannel.id,
});
} else {
await this.gmailFullMessageListFetchService.fetchConnectedAccountThreads(
data.workspaceId,
data.connectedAccountId,
);
}
}
}

View File

@ -1,48 +0,0 @@
import { Injectable, Logger } from '@nestjs/common';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { GoogleAPIRefreshAccessTokenService } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.service';
import { GmailPartialMessageListFetchService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-partial-message-list-fetch.service';
export type GmailPartialMessageListFetchJobData = {
workspaceId: string;
connectedAccountId: string;
};
@Injectable()
export class GmailPartialMessageListFetchJob
implements MessageQueueJob<GmailPartialMessageListFetchJobData>
{
private readonly logger = new Logger(GmailPartialMessageListFetchJob.name);
constructor(
private readonly googleAPIsRefreshAccessTokenService: GoogleAPIRefreshAccessTokenService,
private readonly gmailPartialSyncService: GmailPartialMessageListFetchService,
) {}
async handle(data: GmailPartialMessageListFetchJobData): Promise<void> {
this.logger.log(
`gmail partial-sync for workspace ${data.workspaceId} and account ${data.connectedAccountId}`,
);
try {
await this.googleAPIsRefreshAccessTokenService.refreshAndSaveAccessToken(
data.workspaceId,
data.connectedAccountId,
);
} catch (e) {
this.logger.error(
`Error refreshing access token for connected account ${data.connectedAccountId} in workspace ${data.workspaceId}`,
e,
);
return;
}
await this.gmailPartialSyncService.fetchConnectedAccountThreads(
data.workspaceId,
data.connectedAccountId,
);
}
}

View File

@ -1,75 +0,0 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
import { AutoCompaniesAndContactsCreationModule } from 'src/modules/connected-account/auto-companies-and-contacts-creation/auto-companies-and-contacts-creation.module';
import { GoogleAPIRefreshAccessTokenModule } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.module';
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { BlocklistItemDeleteMessagesJob } from 'src/modules/messaging/jobs/blocklist-item-delete-messages.job';
import { BlocklistReimportMessagesJob } from 'src/modules/messaging/jobs/blocklist-reimport-messages.job';
import { DeleteConnectedAccountAssociatedMessagingDataJob } from 'src/modules/messaging/jobs/delete-connected-account-associated-messaging-data.job';
import { GmailFullMessageListFetchJob } from 'src/modules/messaging/jobs/gmail-full-message-list-fetch.job';
import { GmailMessageListFetchJob } from 'src/modules/messaging/jobs/gmail-message-list-fetch.job';
import { GmailPartialMessageListFetchJob } from 'src/modules/messaging/jobs/gmail-partial-message-list-fetch.job';
import { MessagingCreateCompanyAndContactAfterSyncJob } from 'src/modules/messaging/jobs/messaging-create-company-and-contact-after-sync.job';
import { GmailFullMessageListFetchModule } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch.module';
import { GmailPartialMessageListFetchModule } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-partial-message-list-fetch.module';
import { SetMessageChannelSyncStatusModule } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.module';
import { MessagingTelemetryModule } from 'src/modules/messaging/services/telemetry/messaging-telemetry.module';
import { ThreadCleanerModule } from 'src/modules/messaging/services/thread-cleaner/thread-cleaner.module';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-participant.workspace-entity';
@Module({
imports: [
ObjectMetadataRepositoryModule.forFeature([
ConnectedAccountWorkspaceEntity,
MessageChannelWorkspaceEntity,
MessageParticipantWorkspaceEntity,
MessageChannelMessageAssociationWorkspaceEntity,
BlocklistWorkspaceEntity,
]),
MessagingTelemetryModule,
GmailFullMessageListFetchModule,
GmailPartialMessageListFetchModule,
ThreadCleanerModule,
GoogleAPIRefreshAccessTokenModule,
AutoCompaniesAndContactsCreationModule,
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
SetMessageChannelSyncStatusModule,
],
providers: [
{
provide: BlocklistReimportMessagesJob.name,
useClass: BlocklistReimportMessagesJob,
},
{
provide: BlocklistItemDeleteMessagesJob.name,
useClass: BlocklistItemDeleteMessagesJob,
},
{
provide: GmailFullMessageListFetchJob.name,
useClass: GmailFullMessageListFetchJob,
},
{
provide: GmailPartialMessageListFetchJob.name,
useClass: GmailPartialMessageListFetchJob,
},
{
provide: GmailMessageListFetchJob.name,
useClass: GmailMessageListFetchJob,
},
{
provide: DeleteConnectedAccountAssociatedMessagingDataJob.name,
useClass: DeleteConnectedAccountAssociatedMessagingDataJob,
},
{
provide: MessagingCreateCompanyAndContactAfterSyncJob.name,
useClass: MessagingCreateCompanyAndContactAfterSyncJob,
},
],
})
export class MessagingJobModule {}

View File

@ -2,32 +2,33 @@ import { Injectable, Logger } from '@nestjs/common';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface'; import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { ThreadCleanerService } from 'src/modules/messaging/services/thread-cleaner/thread-cleaner.service'; import { MessagingMessageCleanerService } from 'src/modules/messaging/message-cleaner/services/messaging-message-cleaner.service';
export type DeleteConnectedAccountAssociatedMessagingDataJobData = { export type MessagingConnectedAccountDeletionCleanupJobData = {
workspaceId: string; workspaceId: string;
connectedAccountId: string; connectedAccountId: string;
}; };
@Injectable() @Injectable()
export class DeleteConnectedAccountAssociatedMessagingDataJob export class MessagingConnectedAccountDeletionCleanupJob
implements implements MessageQueueJob<MessagingConnectedAccountDeletionCleanupJobData>
MessageQueueJob<DeleteConnectedAccountAssociatedMessagingDataJobData>
{ {
private readonly logger = new Logger( private readonly logger = new Logger(
DeleteConnectedAccountAssociatedMessagingDataJob.name, MessagingConnectedAccountDeletionCleanupJob.name,
); );
constructor(private readonly threadCleanerService: ThreadCleanerService) {} constructor(
private readonly messageCleanerService: MessagingMessageCleanerService,
) {}
async handle( async handle(
data: DeleteConnectedAccountAssociatedMessagingDataJobData, data: MessagingConnectedAccountDeletionCleanupJobData,
): Promise<void> { ): Promise<void> {
this.logger.log( this.logger.log(
`Deleting connected account ${data.connectedAccountId} associated messaging data in workspace ${data.workspaceId}`, `Deleting connected account ${data.connectedAccountId} associated messaging data in workspace ${data.workspaceId}`,
); );
await this.threadCleanerService.cleanWorkspaceThreads(data.workspaceId); await this.messageCleanerService.cleanWorkspaceThreads(data.workspaceId);
this.logger.log( this.logger.log(
`Deleted connected account ${data.connectedAccountId} associated messaging data in workspace ${data.workspaceId}`, `Deleted connected account ${data.connectedAccountId} associated messaging data in workspace ${data.workspaceId}`,

View File

@ -1,8 +1,5 @@
import { Injectable, Inject } from '@nestjs/common'; import { Inject } from '@nestjs/common';
import { OnEvent } from '@nestjs/event-emitter'; import { OnEvent } from '@nestjs/event-emitter';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { ObjectRecordDeleteEvent } from 'src/engine/integrations/event-emitter/types/object-record-delete.event'; import { ObjectRecordDeleteEvent } from 'src/engine/integrations/event-emitter/types/object-record-delete.event';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants'; import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
@ -11,30 +8,26 @@ import {
DeleteConnectedAccountAssociatedCalendarDataJobData, DeleteConnectedAccountAssociatedCalendarDataJobData,
DeleteConnectedAccountAssociatedCalendarDataJob, DeleteConnectedAccountAssociatedCalendarDataJob,
} from 'src/modules/calendar/jobs/delete-connected-account-associated-calendar-data.job'; } from 'src/modules/calendar/jobs/delete-connected-account-associated-calendar-data.job';
import {
DeleteConnectedAccountAssociatedMessagingDataJobData,
DeleteConnectedAccountAssociatedMessagingDataJob,
} from 'src/modules/messaging/jobs/delete-connected-account-associated-messaging-data.job';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity'; import {
MessagingConnectedAccountDeletionCleanupJob,
MessagingConnectedAccountDeletionCleanupJobData,
} from 'src/modules/messaging/message-cleaner/jobs/messaging-connected-account-deletion-cleanup.job';
@Injectable() export class MessagingMessageCleanerConnectedAccountListener {
export class MessagingConnectedAccountListener {
constructor( constructor(
@Inject(MessageQueue.messagingQueue) @Inject(MessageQueue.messagingQueue)
private readonly messageQueueService: MessageQueueService, private readonly messageQueueService: MessageQueueService,
@Inject(MessageQueue.calendarQueue) @Inject(MessageQueue.calendarQueue)
private readonly calendarQueueService: MessageQueueService, private readonly calendarQueueService: MessageQueueService,
@InjectRepository(FeatureFlagEntity, 'core')
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
) {} ) {}
@OnEvent('connectedAccount.deleted') @OnEvent('connectedAccount.deleted')
async handleDeletedEvent( async handleDeletedEvent(
payload: ObjectRecordDeleteEvent<ConnectedAccountWorkspaceEntity>, payload: ObjectRecordDeleteEvent<ConnectedAccountWorkspaceEntity>,
) { ) {
await this.messageQueueService.add<DeleteConnectedAccountAssociatedMessagingDataJobData>( await this.messageQueueService.add<MessagingConnectedAccountDeletionCleanupJobData>(
DeleteConnectedAccountAssociatedMessagingDataJob.name, MessagingConnectedAccountDeletionCleanupJob.name,
{ {
workspaceId: payload.workspaceId, workspaceId: payload.workspaceId,
connectedAccountId: payload.recordId, connectedAccountId: payload.recordId,

View File

@ -0,0 +1,27 @@
import { Module } from '@nestjs/common';
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
import { MessagingConnectedAccountDeletionCleanupJob } from 'src/modules/messaging/message-cleaner/jobs/messaging-connected-account-deletion-cleanup.job';
import { MessagingMessageCleanerConnectedAccountListener } from 'src/modules/messaging/message-cleaner/listeners/messaging-message-cleaner-connected-account.listener';
import { MessagingMessageCleanerService } from 'src/modules/messaging/message-cleaner/services/messaging-message-cleaner.service';
@Module({
imports: [
ObjectMetadataRepositoryModule.forFeature([
MessageWorkspaceEntity,
MessageThreadWorkspaceEntity,
]),
],
providers: [
MessagingMessageCleanerService,
{
provide: MessagingConnectedAccountDeletionCleanupJob.name,
useClass: MessagingConnectedAccountDeletionCleanupJob,
},
MessagingMessageCleanerConnectedAccountListener,
],
exports: [MessagingMessageCleanerService],
})
export class MessagingMessageCleanerModule {}

View File

@ -1,14 +1,14 @@
import { Injectable } from '@nestjs/common'; import { Injectable } from '@nestjs/common';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { MessageThreadRepository } from 'src/modules/messaging/repositories/message-thread.repository'; import { MessageThreadRepository } from 'src/modules/messaging/common/repositories/message-thread.repository';
import { MessageRepository } from 'src/modules/messaging/repositories/message.repository'; import { MessageRepository } from 'src/modules/messaging/common/repositories/message.repository';
import { deleteUsingPagination } from 'src/modules/messaging/services/thread-cleaner/utils/delete-using-pagination.util'; import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity'; import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity'; import { deleteUsingPagination } from 'src/modules/messaging/message-cleaner/utils/delete-using-pagination.util';
@Injectable() @Injectable()
export class ThreadCleanerService { export class MessagingMessageCleanerService {
constructor( constructor(
@InjectObjectMetadataRepository(MessageWorkspaceEntity) @InjectObjectMetadataRepository(MessageWorkspaceEntity)
private readonly messageRepository: MessageRepository, private readonly messageRepository: MessageRepository,

View File

@ -1,4 +1,4 @@
import { deleteUsingPagination } from './delete-using-pagination.util'; import { deleteUsingPagination } from 'src/modules/messaging/message-cleaner/utils/delete-using-pagination.util';
describe('deleteUsingPagination', () => { describe('deleteUsingPagination', () => {
it('should delete items using pagination', async () => { it('should delete items using pagination', async () => {
@ -29,7 +29,7 @@ describe('deleteUsingPagination', () => {
expect(getterPaginated).toHaveBeenNthCalledWith( expect(getterPaginated).toHaveBeenNthCalledWith(
2, 2,
batchSize, batchSize,
batchSize, 0,
workspaceId, workspaceId,
transactionManager, transactionManager,
); );

View File

@ -0,0 +1 @@
export const MESSAGING_IMPORT_ONGOING_SYNC_TIMEOUT = 1000 * 60 * 60; // 1 hour

View File

@ -4,16 +4,16 @@ import { Command, CommandRunner } from 'nest-commander';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants'; import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service'; import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { GmailMessageListFetchCronJob } from 'src/modules/messaging/crons/jobs/gmail-message-list-fetch.cron.job'; import { MessagingMessageListFetchCronJob } from 'src/modules/messaging/message-import-manager/crons/jobs/messaging-message-list-fetch.cron.job';
const GMAIL_MESSAGE_LIST_FETCH_CRON_PATTERN = '*/5 * * * *'; const MESSAGING_MESSAGE_LIST_FETCH_CRON_PATTERN = '*/5 * * * *';
@Command({ @Command({
name: 'cron:messaging:gmail-message-list-fetch', name: 'cron:messaging:message-list-fetch',
description: description:
'Starts a cron job to sync existing connected account messages and store them in the cache', 'Starts a cron job to sync existing connected account messages and store them in the cache',
}) })
export class GmailMessageListFetchCronCommand extends CommandRunner { export class MessagingMessageListFetchCronCommand extends CommandRunner {
constructor( constructor(
@Inject(MessageQueue.cronQueue) @Inject(MessageQueue.cronQueue)
private readonly messageQueueService: MessageQueueService, private readonly messageQueueService: MessageQueueService,
@ -23,10 +23,10 @@ export class GmailMessageListFetchCronCommand extends CommandRunner {
async run(): Promise<void> { async run(): Promise<void> {
await this.messageQueueService.addCron<undefined>( await this.messageQueueService.addCron<undefined>(
GmailMessageListFetchCronJob.name, MessagingMessageListFetchCronJob.name,
undefined, undefined,
{ {
repeat: { pattern: GMAIL_MESSAGE_LIST_FETCH_CRON_PATTERN }, repeat: { pattern: MESSAGING_MESSAGE_LIST_FETCH_CRON_PATTERN },
}, },
); );
} }

View File

@ -4,13 +4,13 @@ import { Command, CommandRunner } from 'nest-commander';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants'; import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service'; import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { GmailMessagesImportCronJob } from 'src/modules/messaging/crons/jobs/gmail-messages-import.cron.job'; import { MessagingMessagesImportCronJob } from 'src/modules/messaging/message-import-manager/crons/jobs/messaging-messages-import.cron.job';
@Command({ @Command({
name: 'cron:messaging:gmail-messages-import', name: 'cron:messaging:messages-import',
description: 'Starts a cron job to fetch all messages from cache', description: 'Starts a cron job to fetch all messages from cache',
}) })
export class GmailMessagesImportCronCommand extends CommandRunner { export class MessagingMessagesImportCronCommand extends CommandRunner {
constructor( constructor(
@Inject(MessageQueue.cronQueue) @Inject(MessageQueue.cronQueue)
private readonly messageQueueService: MessageQueueService, private readonly messageQueueService: MessageQueueService,
@ -20,7 +20,7 @@ export class GmailMessagesImportCronCommand extends CommandRunner {
async run(): Promise<void> { async run(): Promise<void> {
await this.messageQueueService.addCron<undefined>( await this.messageQueueService.addCron<undefined>(
GmailMessagesImportCronJob.name, MessagingMessagesImportCronJob.name,
undefined, undefined,
{ {
repeat: { repeat: {

View File

@ -7,30 +7,22 @@ import { MessageQueueJob } from 'src/engine/integrations/message-queue/interface
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity'; import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity'; import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity';
import {
GmailPartialMessageListFetchJobData,
GmailPartialMessageListFetchJob,
} from 'src/modules/messaging/jobs/gmail-partial-message-list-fetch.job';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants'; import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service'; import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { EnvironmentService } from 'src/engine/integrations/environment/environment.service'; import { EnvironmentService } from 'src/engine/integrations/environment/environment.service';
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { import {
FeatureFlagEntity, MessagingMessageListFetchJobData,
FeatureFlagKeys, MessagingMessageListFetchJob,
} from 'src/engine/core-modules/feature-flag/feature-flag.entity'; } from 'src/modules/messaging/message-import-manager/jobs/messaging-message-list-fetch.job';
import {
GmailMessageListFetchJobData,
GmailMessageListFetchJob,
} from 'src/modules/messaging/jobs/gmail-message-list-fetch.job';
@Injectable() @Injectable()
export class GmailMessageListFetchCronJob export class MessagingMessageListFetchCronJob
implements MessageQueueJob<undefined> implements MessageQueueJob<undefined>
{ {
private readonly logger = new Logger(GmailMessageListFetchCronJob.name); private readonly logger = new Logger(MessagingMessageListFetchCronJob.name);
constructor( constructor(
@InjectRepository(Workspace, 'core') @InjectRepository(Workspace, 'core')
@ -42,8 +34,6 @@ export class GmailMessageListFetchCronJob
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity) @InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
private readonly messageChannelRepository: MessageChannelRepository, private readonly messageChannelRepository: MessageChannelRepository,
private readonly environmentService: EnvironmentService, private readonly environmentService: EnvironmentService,
@InjectRepository(FeatureFlagEntity, 'core')
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
) {} ) {}
async handle(): Promise<void> { async handle(): Promise<void> {
@ -74,15 +64,6 @@ export class GmailMessageListFetchCronJob
} }
private async enqueueSyncs(workspaceId: string): Promise<void> { private async enqueueSyncs(workspaceId: string): Promise<void> {
const isGmailSyncV2EnabledFeatureFlag =
await this.featureFlagRepository.findOneBy({
workspaceId: workspaceId,
key: FeatureFlagKeys.IsGmailSyncV2Enabled,
value: true,
});
const isGmailSyncV2Enabled = isGmailSyncV2EnabledFeatureFlag?.value;
try { try {
const messageChannels = const messageChannels =
await this.messageChannelRepository.getAll(workspaceId); await this.messageChannelRepository.getAll(workspaceId);
@ -92,23 +73,13 @@ export class GmailMessageListFetchCronJob
continue; continue;
} }
if (isGmailSyncV2Enabled) { await this.messageQueueService.add<MessagingMessageListFetchJobData>(
await this.messageQueueService.add<GmailMessageListFetchJobData>( MessagingMessageListFetchJob.name,
GmailMessageListFetchJob.name, {
{ workspaceId,
workspaceId, connectedAccountId: messageChannel.connectedAccountId,
connectedAccountId: messageChannel.connectedAccountId, },
}, );
);
} else {
await this.messageQueueService.add<GmailPartialMessageListFetchJobData>(
GmailPartialMessageListFetchJob.name,
{
workspaceId,
connectedAccountId: messageChannel.connectedAccountId,
},
);
}
} }
} catch (error) { } catch (error) {
this.logger.error( this.logger.error(

View File

@ -0,0 +1,65 @@
import { Inject, Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository, In } from 'typeorm';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity';
import { EnvironmentService } from 'src/engine/integrations/environment/environment.service';
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
import {
MessagingMessagesImportJobData,
MessagingMessagesImportJob,
} from 'src/modules/messaging/message-import-manager/jobs/messaging-messages-import.job';
@Injectable()
export class MessagingMessagesImportCronJob
implements MessageQueueJob<undefined>
{
private readonly logger = new Logger(MessagingMessagesImportCronJob.name);
constructor(
@InjectRepository(Workspace, 'core')
private readonly workspaceRepository: Repository<Workspace>,
@InjectRepository(DataSourceEntity, 'metadata')
private readonly dataSourceRepository: Repository<DataSourceEntity>,
private readonly environmentService: EnvironmentService,
@Inject(MessageQueue.messagingQueue)
private readonly messageQueueService: MessageQueueService,
) {}
async handle(): Promise<void> {
const workspaceIds = (
await this.workspaceRepository.find({
where: this.environmentService.get('IS_BILLING_ENABLED')
? {
subscriptionStatus: In(['active', 'trialing', 'past_due']),
}
: {},
select: ['id'],
})
).map((workspace) => workspace.id);
const dataSources = await this.dataSourceRepository.find({
where: {
workspaceId: In(workspaceIds),
},
});
const workspaceIdsWithDataSources = new Set(
dataSources.map((dataSource) => dataSource.workspaceId),
);
for (const workspaceId of workspaceIdsWithDataSources) {
await this.messageQueueService.add<MessagingMessagesImportJobData>(
MessagingMessagesImportJob.name,
{
workspaceId,
},
);
}
}
}

View File

@ -0,0 +1,5 @@
export const MESSAGING_GMAIL_EXCLUDED_CATEGORIES = [
'promotions',
'social',
'forums',
];

View File

@ -0,0 +1 @@
export const MESSAGING_GMAIL_USERS_HISTORY_MAX_RESULT = 500;

View File

@ -0,0 +1 @@
export const MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE = 20;

View File

@ -0,0 +1 @@
export const MESSAGING_GMAIL_USERS_MESSAGES_LIST_MAX_RESULT = 500;

View File

@ -0,0 +1,53 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
import { EnvironmentModule } from 'src/engine/integrations/environment/environment.module';
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
import { GoogleAPIRefreshAccessTokenModule } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.module';
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { MessagingGmailClientProvider } from 'src/modules/messaging/message-import-manager/drivers/gmail/providers/messaging-gmail-client.provider';
import { MessagingGmailFetchMessagesByBatchesService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-fetch-messages-by-batches.service';
import { MessagingGmailFetchMessageIdsToExcludeService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-fetch-messages-ids-to-exclude.service';
import { MessagingGmailFullMessageListFetchService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-full-message-list-fetch.service';
import { MessagingGmailHistoryService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-history.service';
import { MessagingGmailMessagesImportService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-messages-import.service';
import { MessagingGmailPartialMessageListFetchService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-partial-message-list-fetch.service';
@Module({
imports: [
GoogleAPIRefreshAccessTokenModule,
EnvironmentModule,
ObjectMetadataRepositoryModule.forFeature([
ConnectedAccountWorkspaceEntity,
MessageChannelWorkspaceEntity,
MessageChannelMessageAssociationWorkspaceEntity,
BlocklistWorkspaceEntity,
]),
MessagingCommonModule,
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
],
providers: [
MessagingGmailClientProvider,
MessagingGmailHistoryService,
MessagingGmailFetchMessagesByBatchesService,
MessagingGmailPartialMessageListFetchService,
MessagingGmailFullMessageListFetchService,
MessagingGmailMessagesImportService,
MessagingGmailFetchMessageIdsToExcludeService,
],
exports: [
MessagingGmailClientProvider,
MessagingGmailHistoryService,
MessagingGmailFetchMessagesByBatchesService,
MessagingGmailPartialMessageListFetchService,
MessagingGmailFullMessageListFetchService,
MessagingGmailMessagesImportService,
MessagingGmailFetchMessageIdsToExcludeService,
],
})
export class MessagingGmailDriverModule {}

View File

@ -6,7 +6,7 @@ import { gmail_v1, google } from 'googleapis';
import { EnvironmentService } from 'src/engine/integrations/environment/environment.service'; import { EnvironmentService } from 'src/engine/integrations/environment/environment.service';
@Injectable() @Injectable()
export class GmailClientProvider { export class MessagingGmailClientProvider {
constructor(private readonly environmentService: EnvironmentService) {} constructor(private readonly environmentService: EnvironmentService) {}
public async getGmailClient(refreshToken: string): Promise<gmail_v1.Gmail> { public async getGmailClient(refreshToken: string): Promise<gmail_v1.Gmail> {

View File

@ -5,17 +5,21 @@ import planer from 'planer';
import addressparser from 'addressparser'; import addressparser from 'addressparser';
import { gmail_v1 } from 'googleapis'; import { gmail_v1 } from 'googleapis';
import { GmailMessage } from 'src/modules/messaging/types/gmail-message';
import { MessageQuery } from 'src/modules/messaging/types/message-or-thread-query';
import { FetchByBatchesService } from 'src/modules/messaging/services/fetch-by-batch/fetch-by-batch.service';
import { formatAddressObjectAsParticipants } from 'src/modules/messaging/services/utils/format-address-object-as-participants.util';
import { assert, assertNotNull } from 'src/utils/assert'; import { assert, assertNotNull } from 'src/utils/assert';
import { GmailMessage } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
import { MessageQuery } from 'src/modules/messaging/message-import-manager/types/message-or-thread-query';
import { formatAddressObjectAsParticipants } from 'src/modules/messaging/message-import-manager/utils/format-address-object-as-participants.util';
import { MessagingFetchByBatchesService } from 'src/modules/messaging/common/services/messaging-fetch-by-batch.service';
@Injectable() @Injectable()
export class FetchMessagesByBatchesService { export class MessagingGmailFetchMessagesByBatchesService {
private readonly logger = new Logger(FetchMessagesByBatchesService.name); private readonly logger = new Logger(
MessagingGmailFetchMessagesByBatchesService.name,
);
constructor(private readonly fetchByBatchesService: FetchByBatchesService) {} constructor(
private readonly fetchByBatchesService: MessagingFetchByBatchesService,
) {}
async fetchAllMessages( async fetchAllMessages(
queries: MessageQuery[], queries: MessageQuery[],

View File

@ -2,15 +2,15 @@ import { Injectable } from '@nestjs/common';
import { gmail_v1 } from 'googleapis'; import { gmail_v1 } from 'googleapis';
import { GMAIL_EXCLUDED_CATEGORIES } from 'src/modules/messaging/constants/gmail-excluded-categories'; import { MESSAGING_GMAIL_EXCLUDED_CATEGORIES } from 'src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-excluded-categories';
import { GmailGetHistoryService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-get-history.service'; import { MessagingGmailHistoryService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-history.service';
import { computeGmailCategoryLabelId } from 'src/modules/messaging/utils/compute-gmail-category-label-id'; import { computeGmailCategoryLabelId } from 'src/modules/messaging/message-import-manager/drivers/gmail/utils/compute-gmail-category-label-id';
import { assertNotNull } from 'src/utils/assert'; import { assertNotNull } from 'src/utils/assert';
@Injectable() @Injectable()
export class GmailFetchMessageIdsToExcludeService { export class MessagingGmailFetchMessageIdsToExcludeService {
constructor( constructor(
private readonly gmailGetHistoryService: GmailGetHistoryService, private readonly gmailGetHistoryService: MessagingGmailHistoryService,
) {} ) {}
public async fetchEmailIdsToExcludeOrThrow( public async fetchEmailIdsToExcludeOrThrow(
@ -19,7 +19,7 @@ export class GmailFetchMessageIdsToExcludeService {
): Promise<string[]> { ): Promise<string[]> {
const emailIds: string[] = []; const emailIds: string[] = [];
for (const category of GMAIL_EXCLUDED_CATEGORIES) { for (const category of MESSAGING_GMAIL_EXCLUDED_CATEGORIES) {
const { history, error } = await this.gmailGetHistoryService.getHistory( const { history, error } = await this.gmailGetHistoryService.getHistory(
gmailClient, gmailClient,
lastSyncHistoryId, lastSyncHistoryId,

View File

@ -9,27 +9,29 @@ import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decora
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum'; import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { GMAIL_USERS_MESSAGES_LIST_MAX_RESULT } from 'src/modules/messaging/constants/gmail-users-messages-list-max-result.constant';
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
import { GmailClientProvider } from 'src/modules/messaging/services/providers/gmail/gmail-client.provider';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { import {
GmailError, GmailError,
GmailErrorHandlingService, MessagingErrorHandlingService,
} from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service'; } from 'src/modules/messaging/common/services/messaging-error-handling.service';
import { MessageChannelSyncStatusService } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.service'; import { computeGmailCategoryExcludeSearchFilter } from 'src/modules/messaging/message-import-manager/drivers/gmail/utils/compute-gmail-category-excude-search-filter';
import { computeGmailCategoryExcludeSearchFilter } from 'src/modules/messaging/utils/compute-gmail-category-excude-search-filter'; import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
import { GMAIL_EXCLUDED_CATEGORIES } from 'src/modules/messaging/constants/gmail-excluded-categories'; import { MessagingGmailClientProvider } from 'src/modules/messaging/message-import-manager/drivers/gmail/providers/messaging-gmail-client.provider';
import { MESSAGING_GMAIL_USERS_MESSAGES_LIST_MAX_RESULT } from 'src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-users-messages-list-max-result.constant';
import { MESSAGING_GMAIL_EXCLUDED_CATEGORIES } from 'src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-excluded-categories';
@Injectable() @Injectable()
export class GmailFullMessageListFetchV2Service { export class MessagingGmailFullMessageListFetchService {
private readonly logger = new Logger(GmailFullMessageListFetchV2Service.name); private readonly logger = new Logger(
MessagingGmailFullMessageListFetchService.name,
);
constructor( constructor(
private readonly gmailClientProvider: GmailClientProvider, private readonly gmailClientProvider: MessagingGmailClientProvider,
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity) @InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
private readonly messageChannelRepository: MessageChannelRepository, private readonly messageChannelRepository: MessageChannelRepository,
@InjectCacheStorage(CacheStorageNamespace.Messaging) @InjectCacheStorage(CacheStorageNamespace.Messaging)
@ -38,8 +40,8 @@ export class GmailFullMessageListFetchV2Service {
MessageChannelMessageAssociationWorkspaceEntity, MessageChannelMessageAssociationWorkspaceEntity,
) )
private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository, private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository,
private readonly messageChannelSyncStatusService: MessageChannelSyncStatusService, private readonly messagingChannelSyncStatusService: MessagingChannelSyncStatusService,
private readonly gmailErrorHandlingService: GmailErrorHandlingService, private readonly gmailErrorHandlingService: MessagingErrorHandlingService,
) {} ) {}
public async processMessageListFetch( public async processMessageListFetch(
@ -47,7 +49,7 @@ export class GmailFullMessageListFetchV2Service {
connectedAccount: ObjectRecord<ConnectedAccountWorkspaceEntity>, connectedAccount: ObjectRecord<ConnectedAccountWorkspaceEntity>,
workspaceId: string, workspaceId: string,
) { ) {
await this.messageChannelSyncStatusService.markAsMessagesListFetchOngoing( await this.messagingChannelSyncStatusService.markAsMessagesListFetchOngoing(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
@ -75,7 +77,7 @@ export class GmailFullMessageListFetchV2Service {
return; return;
} }
await this.messageChannelSyncStatusService.scheduleMessagesImport( await this.messagingChannelSyncStatusService.scheduleMessagesImport(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
@ -97,9 +99,11 @@ export class GmailFullMessageListFetchV2Service {
try { try {
response = await gmailClient.users.messages.list({ response = await gmailClient.users.messages.list({
userId: 'me', userId: 'me',
maxResults: GMAIL_USERS_MESSAGES_LIST_MAX_RESULT, maxResults: MESSAGING_GMAIL_USERS_MESSAGES_LIST_MAX_RESULT,
pageToken, pageToken,
q: computeGmailCategoryExcludeSearchFilter(GMAIL_EXCLUDED_CATEGORIES), q: computeGmailCategoryExcludeSearchFilter(
MESSAGING_GMAIL_EXCLUDED_CATEGORIES,
),
}); });
} catch (error) { } catch (error) {
return { return {

View File

@ -3,11 +3,11 @@ import { Injectable } from '@nestjs/common';
import { gmail_v1 } from 'googleapis'; import { gmail_v1 } from 'googleapis';
import { GaxiosResponse } from 'gaxios'; import { GaxiosResponse } from 'gaxios';
import { GMAIL_USERS_HISTORY_MAX_RESULT } from 'src/modules/messaging/constants/gmail-users-history-max-result.constant'; import { GmailError } from 'src/modules/messaging/common/services/messaging-error-handling.service';
import { GmailError } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service'; import { MESSAGING_GMAIL_USERS_HISTORY_MAX_RESULT } from 'src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-users-history-max-result.constant';
@Injectable() @Injectable()
export class GmailGetHistoryService { export class MessagingGmailHistoryService {
constructor() {} constructor() {}
public async getHistory( public async getHistory(
@ -30,7 +30,7 @@ export class GmailGetHistoryService {
try { try {
response = await gmailClient.users.history.list({ response = await gmailClient.users.history.list({
userId: 'me', userId: 'me',
maxResults: GMAIL_USERS_HISTORY_MAX_RESULT, maxResults: MESSAGING_GMAIL_USERS_HISTORY_MAX_RESULT,
pageToken, pageToken,
startHistoryId: lastSyncHistoryId, startHistoryId: lastSyncHistoryId,
historyTypes: historyTypes || ['messageAdded', 'messageDeleted'], historyTypes: historyTypes || ['messageAdded', 'messageDeleted'],

View File

@ -1,39 +1,40 @@
import { Injectable, Logger } from '@nestjs/common'; import { Injectable, Logger } from '@nestjs/common';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { FetchMessagesByBatchesService } from 'src/modules/messaging/services/fetch-messages-by-batches/fetch-messages-by-batches.service';
import {
MessageChannelWorkspaceEntity,
MessageChannelSyncSubStatus,
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { createQueriesFromMessageIds } from 'src/modules/messaging/utils/create-queries-from-message-ids.util';
import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator'; import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator';
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum'; import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache-storage.service'; import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache-storage.service';
import { GMAIL_USERS_MESSAGES_GET_BATCH_SIZE } from 'src/modules/messaging/constants/gmail-users-messages-get-batch-size.constant';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { SaveMessagesAndEnqueueContactCreationService } from 'src/modules/messaging/services/gmail-messages-import/save-messages-and-enqueue-contact-creation.service';
import { GmailErrorHandlingService } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service';
import { MessageChannelSyncStatusService } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.service';
import { GoogleAPIRefreshAccessTokenService } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.service'; import { GoogleAPIRefreshAccessTokenService } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.service';
import { GmailMessagesImportService } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import.service';
import { MessagingTelemetryService } from 'src/modules/messaging/services/telemetry/messaging-telemetry.service';
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity'; import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { BlocklistRepository } from 'src/modules/connected-account/repositories/blocklist.repository'; import { BlocklistRepository } from 'src/modules/connected-account/repositories/blocklist.repository';
import { filterEmails } from 'src/modules/messaging/services/utils/filter-emails.util'; import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
import {
MessageChannelWorkspaceEntity,
MessageChannelSyncSubStatus,
} from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { createQueriesFromMessageIds } from 'src/modules/messaging/message-import-manager/utils/create-queries-from-message-ids.util';
import { filterEmails } from 'src/modules/messaging/message-import-manager/utils/filter-emails.util';
import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
import { MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE } from 'src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-users-messages-get-batch-size.constant';
import { MessagingGmailFetchMessagesByBatchesService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-fetch-messages-by-batches.service';
import { MessagingErrorHandlingService } from 'src/modules/messaging/common/services/messaging-error-handling.service';
import { MessagingSaveMessagesAndEnqueueContactCreationService } from 'src/modules/messaging/common/services/messaging-save-messages-and-enqueue-contact-creation.service';
@Injectable() @Injectable()
export class GmailMessagesImportV2Service { export class MessagingGmailMessagesImportService {
private readonly logger = new Logger(GmailMessagesImportService.name); private readonly logger = new Logger(
MessagingGmailMessagesImportService.name,
);
constructor( constructor(
private readonly fetchMessagesByBatchesService: FetchMessagesByBatchesService, private readonly fetchMessagesByBatchesService: MessagingGmailFetchMessagesByBatchesService,
@InjectCacheStorage(CacheStorageNamespace.Messaging) @InjectCacheStorage(CacheStorageNamespace.Messaging)
private readonly cacheStorage: CacheStorageService, private readonly cacheStorage: CacheStorageService,
private readonly messageChannelSyncStatusService: MessageChannelSyncStatusService, private readonly messagingChannelSyncStatusService: MessagingChannelSyncStatusService,
private readonly saveMessagesAndEnqueueContactCreationService: SaveMessagesAndEnqueueContactCreationService, private readonly saveMessagesAndEnqueueContactCreationService: MessagingSaveMessagesAndEnqueueContactCreationService,
private readonly gmailErrorHandlingService: GmailErrorHandlingService, private readonly gmailErrorHandlingService: MessagingErrorHandlingService,
private readonly googleAPIsRefreshAccessTokenService: GoogleAPIRefreshAccessTokenService, private readonly googleAPIsRefreshAccessTokenService: GoogleAPIRefreshAccessTokenService,
private readonly messagingTelemetryService: MessagingTelemetryService, private readonly messagingTelemetryService: MessagingTelemetryService,
@InjectObjectMetadataRepository(BlocklistWorkspaceEntity) @InjectObjectMetadataRepository(BlocklistWorkspaceEntity)
@ -63,7 +64,7 @@ export class GmailMessagesImportV2Service {
`Messaging import for workspace ${workspaceId} and account ${connectedAccount.id} starting...`, `Messaging import for workspace ${workspaceId} and account ${connectedAccount.id} starting...`,
); );
await this.messageChannelSyncStatusService.markAsMessagesImportOngoing( await this.messagingChannelSyncStatusService.markAsMessagesImportOngoing(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
@ -76,11 +77,11 @@ export class GmailMessagesImportV2Service {
const messageIdsToFetch = const messageIdsToFetch =
(await this.cacheStorage.setPop( (await this.cacheStorage.setPop(
`messages-to-import:${workspaceId}:gmail:${messageChannel.id}`, `messages-to-import:${workspaceId}:gmail:${messageChannel.id}`,
GMAIL_USERS_MESSAGES_GET_BATCH_SIZE, MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE,
)) ?? []; )) ?? [];
if (!messageIdsToFetch?.length) { if (!messageIdsToFetch?.length) {
await this.messageChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch( await this.messagingChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
@ -119,13 +120,15 @@ export class GmailMessagesImportV2Service {
workspaceId, workspaceId,
); );
if (messageIdsToFetch.length < GMAIL_USERS_MESSAGES_GET_BATCH_SIZE) { if (
await this.messageChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch( messageIdsToFetch.length < MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE
) {
await this.messagingChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
} else { } else {
await this.messageChannelSyncStatusService.scheduleMessagesImport( await this.messagingChannelSyncStatusService.scheduleMessagesImport(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );

View File

@ -2,30 +2,30 @@ import { Injectable, Logger } from '@nestjs/common';
import { gmail_v1 } from 'googleapis'; import { gmail_v1 } from 'googleapis';
import { GmailClientProvider } from 'src/modules/messaging/services/providers/gmail/gmail-client.provider';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache-storage.service'; import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache-storage.service';
import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator'; import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator';
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum'; import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
import { GmailGetHistoryService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-get-history.service';
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record'; import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
import { GmailErrorHandlingService } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service'; import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
import { MessageChannelSyncStatusService } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.service'; import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { GmailFetchMessageIdsToExcludeService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-fetch-messages-ids-to-exclude.service'; import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { MessagingGmailClientProvider } from 'src/modules/messaging/message-import-manager/drivers/gmail/providers/messaging-gmail-client.provider';
import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
import { MessagingGmailFetchMessageIdsToExcludeService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-fetch-messages-ids-to-exclude.service';
import { MessagingErrorHandlingService } from 'src/modules/messaging/common/services/messaging-error-handling.service';
import { MessagingGmailHistoryService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-history.service';
@Injectable() @Injectable()
export class GmailPartialMessageListFetchV2Service { export class MessagingGmailPartialMessageListFetchService {
private readonly logger = new Logger( private readonly logger = new Logger(
GmailPartialMessageListFetchV2Service.name, MessagingGmailPartialMessageListFetchService.name,
); );
constructor( constructor(
private readonly gmailClientProvider: GmailClientProvider, private readonly gmailClientProvider: MessagingGmailClientProvider,
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity) @InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
private readonly messageChannelRepository: MessageChannelRepository, private readonly messageChannelRepository: MessageChannelRepository,
@InjectCacheStorage(CacheStorageNamespace.Messaging) @InjectCacheStorage(CacheStorageNamespace.Messaging)
@ -34,10 +34,10 @@ export class GmailPartialMessageListFetchV2Service {
MessageChannelMessageAssociationWorkspaceEntity, MessageChannelMessageAssociationWorkspaceEntity,
) )
private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository, private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository,
private readonly gmailErrorHandlingService: GmailErrorHandlingService, private readonly gmailErrorHandlingService: MessagingErrorHandlingService,
private readonly gmailGetHistoryService: GmailGetHistoryService, private readonly gmailGetHistoryService: MessagingGmailHistoryService,
private readonly messageChannelSyncStatusService: MessageChannelSyncStatusService, private readonly messagingChannelSyncStatusService: MessagingChannelSyncStatusService,
private readonly gmailFetchMessageIdsToExcludeService: GmailFetchMessageIdsToExcludeService, private readonly gmailFetchMessageIdsToExcludeService: MessagingGmailFetchMessageIdsToExcludeService,
) {} ) {}
public async processMessageListFetch( public async processMessageListFetch(
@ -45,7 +45,7 @@ export class GmailPartialMessageListFetchV2Service {
connectedAccount: ObjectRecord<ConnectedAccountWorkspaceEntity>, connectedAccount: ObjectRecord<ConnectedAccountWorkspaceEntity>,
workspaceId: string, workspaceId: string,
): Promise<void> { ): Promise<void> {
await this.messageChannelSyncStatusService.markAsMessagesListFetchOngoing( await this.messagingChannelSyncStatusService.markAsMessagesListFetchOngoing(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
@ -85,7 +85,7 @@ export class GmailPartialMessageListFetchV2Service {
`Partial message list import done with history ${historyId} and nothing to update for workspace ${workspaceId} and account ${connectedAccount.id}`, `Partial message list import done with history ${historyId} and nothing to update for workspace ${workspaceId} and account ${connectedAccount.id}`,
); );
await this.messageChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch( await this.messagingChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );
@ -144,7 +144,7 @@ export class GmailPartialMessageListFetchV2Service {
workspaceId, workspaceId,
); );
await this.messageChannelSyncStatusService.scheduleMessagesImport( await this.messagingChannelSyncStatusService.scheduleMessagesImport(
messageChannel.id, messageChannel.id,
workspaceId, workspaceId,
); );

View File

@ -1,4 +1,4 @@
import { computeGmailCategoryExcludeSearchFilter } from 'src/modules/messaging/utils/compute-gmail-category-excude-search-filter'; import { computeGmailCategoryExcludeSearchFilter } from 'src/modules/messaging/message-import-manager/drivers/gmail/utils/compute-gmail-category-excude-search-filter';
describe('computeGmailCategoryExcludeSearchFilter', () => { describe('computeGmailCategoryExcludeSearchFilter', () => {
it('should return correct exclude search filter with empty category array', () => { it('should return correct exclude search filter with empty category array', () => {

View File

@ -1,4 +1,4 @@
import { computeGmailCategoryLabelId } from 'src/modules/messaging/utils/compute-gmail-category-label-id'; import { computeGmailCategoryLabelId } from 'src/modules/messaging/message-import-manager/drivers/gmail/utils/compute-gmail-category-label-id';
describe('computeGmailCategoryLabelId', () => { describe('computeGmailCategoryLabelId', () => {
it('should return correct category label id', () => { it('should return correct category label id', () => {

View File

@ -2,32 +2,32 @@ import { Injectable, Logger } from '@nestjs/common';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface'; import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { GmailPartialMessageListFetchV2Service } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-partial-message-list-fetch-v2.service';
import {
MessageChannelSyncSubStatus,
MessageChannelWorkspaceEntity,
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
import { GmailFullMessageListFetchV2Service } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch-v2.service';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator'; import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository'; import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity'; import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository'; import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { MessagingTelemetryService } from 'src/modules/messaging/services/telemetry/messaging-telemetry.service'; import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
import {
MessageChannelSyncSubStatus,
MessageChannelWorkspaceEntity,
} from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { MessagingGmailFullMessageListFetchService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-full-message-list-fetch.service';
import { MessagingGmailPartialMessageListFetchService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-partial-message-list-fetch.service';
export type GmailMessageListFetchJobData = { export type MessagingMessageListFetchJobData = {
workspaceId: string; workspaceId: string;
connectedAccountId: string; connectedAccountId: string;
}; };
@Injectable() @Injectable()
export class GmailMessageListFetchJob export class MessagingMessageListFetchJob
implements MessageQueueJob<GmailMessageListFetchJobData> implements MessageQueueJob<MessagingMessageListFetchJobData>
{ {
private readonly logger = new Logger(GmailMessageListFetchJob.name); private readonly logger = new Logger(MessagingMessageListFetchJob.name);
constructor( constructor(
private readonly gmailFullMessageListFetchV2Service: GmailFullMessageListFetchV2Service, private readonly gmailFullMessageListFetchService: MessagingGmailFullMessageListFetchService,
private readonly gmailPartialMessageListFetchV2Service: GmailPartialMessageListFetchV2Service, private readonly gmailPartialMessageListFetchV2Service: MessagingGmailPartialMessageListFetchService,
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity) @InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
private readonly connectedAccountRepository: ConnectedAccountRepository, private readonly connectedAccountRepository: ConnectedAccountRepository,
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity) @InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
@ -35,7 +35,7 @@ export class GmailMessageListFetchJob
private readonly messagingTelemetryService: MessagingTelemetryService, private readonly messagingTelemetryService: MessagingTelemetryService,
) {} ) {}
async handle(data: GmailMessageListFetchJobData): Promise<void> { async handle(data: MessagingMessageListFetchJobData): Promise<void> {
const { workspaceId, connectedAccountId } = data; const { workspaceId, connectedAccountId } = data;
await this.messagingTelemetryService.track({ await this.messagingTelemetryService.track({
@ -113,7 +113,7 @@ export class GmailMessageListFetchJob
connectedAccountId, connectedAccountId,
}); });
await this.gmailFullMessageListFetchV2Service.processMessageListFetch( await this.gmailFullMessageListFetchService.processMessageListFetch(
messageChannel, messageChannel,
connectedAccount, connectedAccount,
workspaceId, workspaceId,

View File

@ -0,0 +1,61 @@
import { Injectable } from '@nestjs/common';
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
import { MessagingGmailMessagesImportService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-messages-import.service';
export type MessagingMessagesImportJobData = {
workspaceId: string;
};
@Injectable()
export class MessagingMessagesImportJob
implements MessageQueueJob<MessagingMessagesImportJobData>
{
constructor(
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
private readonly connectedAccountRepository: ConnectedAccountRepository,
private readonly gmailFetchMessageContentFromCacheService: MessagingGmailMessagesImportService,
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
private readonly messageChannelRepository: MessageChannelRepository,
private readonly messagingTelemetryService: MessagingTelemetryService,
) {}
async handle(data: MessagingMessagesImportJobData): Promise<void> {
const { workspaceId } = data;
const messageChannels =
await this.messageChannelRepository.getAll(workspaceId);
for (const messageChannel of messageChannels) {
if (!messageChannel?.isSyncEnabled) {
continue;
}
await this.messagingTelemetryService.track({
eventName: 'messages_import.triggered',
workspaceId,
connectedAccountId: messageChannel.connectedAccountId,
messageChannelId: messageChannel.id,
});
const connectedAccount =
await this.connectedAccountRepository.getConnectedAccountOrThrow(
workspaceId,
messageChannel.connectedAccountId,
);
await this.gmailFetchMessageContentFromCacheService.processMessageBatchImport(
messageChannel,
connectedAccount,
workspaceId,
);
}
}
}

View File

@ -0,0 +1,44 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity';
import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
import { MessagingMessageListFetchCronCommand } from 'src/modules/messaging/message-import-manager/crons/commands/messaging-message-list-fetch.cron.command';
import { MessagingMessagesImportCronCommand } from 'src/modules/messaging/message-import-manager/crons/commands/messaging-messages-import.cron.command';
import { MessagingMessageListFetchCronJob } from 'src/modules/messaging/message-import-manager/crons/jobs/messaging-message-list-fetch.cron.job';
import { MessagingMessagesImportCronJob } from 'src/modules/messaging/message-import-manager/crons/jobs/messaging-messages-import.cron.job';
import { MessagingGmailDriverModule } from 'src/modules/messaging/message-import-manager/drivers/gmail/messaging-gmail-driver.module';
import { MessagingMessageListFetchJob } from 'src/modules/messaging/message-import-manager/jobs/messaging-message-list-fetch.job';
import { MessagingMessagesImportJob } from 'src/modules/messaging/message-import-manager/jobs/messaging-messages-import.job';
@Module({
imports: [
MessagingGmailDriverModule,
MessagingCommonModule,
TypeOrmModule.forFeature([Workspace], 'core'),
TypeOrmModule.forFeature([DataSourceEntity], 'metadata'),
],
providers: [
MessagingMessageListFetchCronCommand,
MessagingMessagesImportCronCommand,
{
provide: MessagingMessageListFetchJob.name,
useClass: MessagingMessageListFetchJob,
},
{
provide: MessagingMessagesImportJob.name,
useClass: MessagingMessagesImportJob,
},
{
provide: MessagingMessageListFetchCronJob.name,
useClass: MessagingMessageListFetchCronJob,
},
{
provide: MessagingMessagesImportCronJob.name,
useClass: MessagingMessagesImportCronJob,
},
],
exports: [],
})
export class MessagingImportManagerModule {}

View File

@ -1,4 +1,4 @@
import { formatAddressObjectAsParticipants } from 'src/modules/messaging/services/utils/format-address-object-as-participants.util'; import { formatAddressObjectAsParticipants } from 'src/modules/messaging/message-import-manager/utils/format-address-object-as-participants.util';
describe('formatAddressObjectAsParticipants', () => { describe('formatAddressObjectAsParticipants', () => {
it('should format address object as participants', () => { it('should format address object as participants', () => {

View File

@ -1,4 +1,4 @@
import { MessageQuery } from 'src/modules/messaging/types/message-or-thread-query'; import { MessageQuery } from 'src/modules/messaging/message-import-manager/types/message-or-thread-query';
export const createQueriesFromMessageIds = ( export const createQueriesFromMessageIds = (
messageExternalIds: string[], messageExternalIds: string[],

View File

@ -1,5 +1,5 @@
import { GmailMessage } from 'src/modules/messaging/types/gmail-message';
import { isEmailBlocklisted } from 'src/modules/calendar-messaging-participant/utils/is-email-blocklisted.util'; import { isEmailBlocklisted } from 'src/modules/calendar-messaging-participant/utils/is-email-blocklisted.util';
import { GmailMessage } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
// Todo: refactor this into several utils // Todo: refactor this into several utils
export const filterEmails = (messages: GmailMessage[], blocklist: string[]) => { export const filterEmails = (messages: GmailMessage[], blocklist: string[]) => {

View File

@ -1,6 +1,6 @@
import addressparser from 'addressparser'; import addressparser from 'addressparser';
import { Participant } from 'src/modules/messaging/types/gmail-message'; import { Participant } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
const formatAddressObjectAsArray = ( const formatAddressObjectAsArray = (
addressObject: addressparser.EmailAddress | addressparser.EmailAddress[], addressObject: addressparser.EmailAddress | addressparser.EmailAddress[],

Some files were not shown because too many files have changed in this diff Show More