Rework messaging modules (#5710)
In this PR, I'm refactoring the messaging module into smaller pieces that have **ONE** responsibility: import messages, clean messages, handle message participant creation, instead of having ~30 modules (1 per service, jobs, cron, ...). This is mandatory to start introducing drivers (gmails, office365, ...) IMO. It is too difficult to enforce common interfaces as we have too many interfaces (30 modules...). All modules should not be exposed Right now, we have services that are almost functions: do-that-and-this.service.ts / do-that-and-this.module.ts I believe we should have something more organized at a high level and it does not matter that much if we have a bit of code duplicates. Note that the proposal is not fully implemented in the current PR that has only focused on messaging folder (biggest part) Here is the high level proposal: - connected-account: token-refresher - blocklist - messaging: message-importer, message-cleaner, message-participants, ... (right now I'm keeping a big messaging-common but this will disappear see below) - calendar: calendar-importer, calendar-cleaner, ... Consequences: 1) It's OK to re-implement several times some things. Example: - error handling in connected-account, messaging, and calendar instead of trying to unify. They are actually different error handling. The only things that might be in common is the GmailError => CommonError parsing and I'm not even sure it makes a lot of sense as these 3 apis might have different format actually - auto-creation. Calendar and Messaging could actually have different rules 2) **We should not have circular dependencies:** - I believe this was the reason why we had so many modules, to be able to cherry pick the one we wanted to avoid circular deps. This is not the right approach IMO, we need architect the whole messaging by defining high level blocks that won't have circular dependencies by design. If we encounter one, we should rethink and break the block in a way that makes sense. - ex: connected-account.resolver is not in the same module as token-refresher. ==> connected-account.resolver => message-importer (as we trigger full sync job when we connect an account) => token-refresher (as we refresh token on message import). connected-account.resolver and token-refresher both in connected-account folder but should be in different modules. Otherwise it's a circular dependency. It does not mean that we should create 1 module per service as it was done before In a nutshell: The code needs to be thought in term of reponsibilities and in a way that enforce high level interfaces (and avoid circular dependencies) Bonus: As you can see, this code is also removing a lot of code because of the removal of many .module.ts (also because I'm removing the sync scripts v2 feature flag end removing old code) Bonus: I have prefixed services name with Messaging to improve dev xp. GmailErrorHandler could be different between MessagingGmailErrorHandler and CalendarGmailErrorHandler for instance
This commit is contained in:
Charles Bochet
@ -3,10 +3,10 @@ import { Module } from '@nestjs/common';
|
||||
import { MatchParticipantJob } from 'src/modules/calendar-messaging-participant/jobs/match-participant.job';
|
||||
import { UnmatchParticipantJob } from 'src/modules/calendar-messaging-participant/jobs/unmatch-participant.job';
|
||||
import { CalendarEventParticipantModule } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.module';
|
||||
import { MessageParticipantModule } from 'src/modules/messaging/services/message-participant/message-participant.module';
|
||||
import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
|
||||
|
||||
@Module({
|
||||
imports: [MessageParticipantModule, CalendarEventParticipantModule],
|
||||
imports: [CalendarEventParticipantModule, MessagingCommonModule],
|
||||
providers: [
|
||||
{
|
||||
provide: MatchParticipantJob.name,
|
||||
|
||||
@ -3,7 +3,7 @@ import { Injectable } from '@nestjs/common';
|
||||
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
|
||||
|
||||
import { CalendarEventParticipantService } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.service';
|
||||
import { MessageParticipantService } from 'src/modules/messaging/services/message-participant/message-participant.service';
|
||||
import { MessagingMessageParticipantService } from 'src/modules/messaging/common/services/messaging-message-participant.service';
|
||||
|
||||
export type MatchParticipantJobData = {
|
||||
workspaceId: string;
|
||||
@ -17,7 +17,7 @@ export class MatchParticipantJob
|
||||
implements MessageQueueJob<MatchParticipantJobData>
|
||||
{
|
||||
constructor(
|
||||
private readonly messageParticipantService: MessageParticipantService,
|
||||
private readonly messageParticipantService: MessagingMessageParticipantService,
|
||||
private readonly calendarEventParticipantService: CalendarEventParticipantService,
|
||||
) {}
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@ import { Injectable } from '@nestjs/common';
|
||||
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
|
||||
|
||||
import { CalendarEventParticipantService } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.service';
|
||||
import { MessageParticipantService } from 'src/modules/messaging/services/message-participant/message-participant.service';
|
||||
import { MessagingMessageParticipantService } from 'src/modules/messaging/common/services/messaging-message-participant.service';
|
||||
|
||||
export type UnmatchParticipantJobData = {
|
||||
workspaceId: string;
|
||||
@ -17,7 +17,7 @@ export class UnmatchParticipantJob
|
||||
implements MessageQueueJob<UnmatchParticipantJobData>
|
||||
{
|
||||
constructor(
|
||||
private readonly messageParticipantService: MessageParticipantService,
|
||||
private readonly messageParticipantService: MessagingMessageParticipantService,
|
||||
private readonly calendarEventParticipantService: CalendarEventParticipantService,
|
||||
) {}
|
||||
|
||||
|
||||
@ -9,7 +9,7 @@ import {
|
||||
CalendarCreateCompanyAndContactAfterSyncJobData,
|
||||
CalendarCreateCompanyAndContactAfterSyncJob,
|
||||
} from 'src/modules/calendar/jobs/calendar-create-company-and-contact-after-sync.job';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
|
||||
@Injectable()
|
||||
export class CalendarChannelListener {
|
||||
|
||||
@ -3,7 +3,7 @@ import { Injectable } from '@nestjs/common';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { CalendarEventRepository } from 'src/modules/calendar/repositories/calendar-event.repository';
|
||||
import { CalendarEventWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-event.workspace-entity';
|
||||
import { deleteUsingPagination } from 'src/modules/messaging/services/thread-cleaner/utils/delete-using-pagination.util';
|
||||
import { deleteUsingPagination } from 'src/modules/messaging/message-cleaner/utils/delete-using-pagination.util';
|
||||
|
||||
@Injectable()
|
||||
export class CalendarEventCleanerService {
|
||||
|
||||
@ -7,11 +7,11 @@ import { CreateContactModule } from 'src/modules/connected-account/auto-companie
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity';
|
||||
import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity';
|
||||
import { MessageParticipantModule } from 'src/modules/messaging/services/message-participant/message-participant.module';
|
||||
import { WorkspaceDataSourceModule } from 'src/engine/workspace-datasource/workspace-datasource.module';
|
||||
import { CalendarEventParticipantWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-event-participant.workspace-entity';
|
||||
import { CalendarEventParticipantModule } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.module';
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
@ -22,7 +22,7 @@ import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-
|
||||
WorkspaceMemberWorkspaceEntity,
|
||||
CalendarEventParticipantWorkspaceEntity,
|
||||
]),
|
||||
MessageParticipantModule,
|
||||
MessagingCommonModule,
|
||||
WorkspaceDataSourceModule,
|
||||
CalendarEventParticipantModule,
|
||||
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
|
||||
|
||||
@ -5,11 +5,11 @@ import { ObjectRecordUpdateEvent } from 'src/engine/integrations/event-emitter/t
|
||||
import { objectRecordChangedProperties } from 'src/engine/integrations/event-emitter/utils/object-record-changed-properties.util';
|
||||
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
|
||||
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import {
|
||||
MessagingCreateCompanyAndContactAfterSyncJobData,
|
||||
MessagingCreateCompanyAndContactAfterSyncJob,
|
||||
} from 'src/modules/messaging/jobs/messaging-create-company-and-contact-after-sync.job';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
} from 'src/modules/messaging/message-participants-manager/jobs/messaging-create-company-and-contact-after-sync.job';
|
||||
|
||||
@Injectable()
|
||||
export class MessagingMessageChannelListener {
|
||||
@ -1,7 +1,6 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { EntityManager, Repository } from 'typeorm';
|
||||
import { EntityManager } from 'typeorm';
|
||||
import compact from 'lodash.compact';
|
||||
|
||||
import { getDomainNameFromHandle } from 'src/modules/calendar-messaging-participant/utils/get-domain-name-from-handle.util';
|
||||
@ -16,11 +15,10 @@ import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/sta
|
||||
import { getUniqueContactsAndHandles } from 'src/modules/connected-account/auto-companies-and-contacts-creation/utils/get-unique-contacts-and-handles.util';
|
||||
import { Contacts } from 'src/modules/connected-account/auto-companies-and-contacts-creation/types/contact.type';
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import { MessageParticipantService } from 'src/modules/messaging/services/message-participant/message-participant.service';
|
||||
import { CalendarEventParticipantService } from 'src/modules/calendar/services/calendar-event-participant/calendar-event-participant.service';
|
||||
import { filterOutContactsFromCompanyOrWorkspace } from 'src/modules/connected-account/auto-companies-and-contacts-creation/utils/filter-out-contacts-from-company-or-workspace.util';
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { MessagingMessageParticipantService } from 'src/modules/messaging/common/services/messaging-message-participant.service';
|
||||
|
||||
@Injectable()
|
||||
export class CreateCompanyAndContactService {
|
||||
@ -32,10 +30,8 @@ export class CreateCompanyAndContactService {
|
||||
@InjectObjectMetadataRepository(WorkspaceMemberWorkspaceEntity)
|
||||
private readonly workspaceMemberRepository: WorkspaceMemberRepository,
|
||||
private readonly workspaceDataSourceService: WorkspaceDataSourceService,
|
||||
private readonly messageParticipantService: MessageParticipantService,
|
||||
private readonly messageParticipantService: MessagingMessageParticipantService,
|
||||
private readonly calendarEventParticipantService: CalendarEventParticipantService,
|
||||
@InjectRepository(FeatureFlagEntity, 'core')
|
||||
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
|
||||
) {}
|
||||
|
||||
async createCompaniesAndPeople(
|
||||
|
||||
@ -3,14 +3,14 @@ import { Module } from '@nestjs/common';
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { GoogleAPIRefreshAccessTokenService } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.service';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { GmailErrorHandlingModule } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.module';
|
||||
import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
ObjectMetadataRepositoryModule.forFeature([
|
||||
ConnectedAccountWorkspaceEntity,
|
||||
]),
|
||||
GmailErrorHandlingModule,
|
||||
MessagingCommonModule,
|
||||
],
|
||||
providers: [GoogleAPIRefreshAccessTokenService],
|
||||
exports: [GoogleAPIRefreshAccessTokenService],
|
||||
|
||||
@ -6,9 +6,10 @@ import { EnvironmentService } from 'src/engine/integrations/environment/environm
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { GmailErrorHandlingService } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
|
||||
|
||||
@Injectable()
|
||||
export class GoogleAPIRefreshAccessTokenService {
|
||||
@ -18,7 +19,8 @@ export class GoogleAPIRefreshAccessTokenService {
|
||||
private readonly connectedAccountRepository: ConnectedAccountRepository,
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
private readonly gmailErrorHandlingService: GmailErrorHandlingService,
|
||||
private readonly messagingTelemetryService: MessagingTelemetryService,
|
||||
private readonly messagingChannelSyncStatusService: MessagingChannelSyncStatusService,
|
||||
) {}
|
||||
|
||||
async refreshAndSaveAccessToken(
|
||||
@ -65,13 +67,21 @@ export class GoogleAPIRefreshAccessTokenService {
|
||||
);
|
||||
}
|
||||
|
||||
await this.gmailErrorHandlingService.handleGmailError(
|
||||
{
|
||||
code: error.code,
|
||||
reason: error.response.data.error,
|
||||
},
|
||||
'messages-import',
|
||||
messageChannel,
|
||||
await this.messagingTelemetryService.track({
|
||||
eventName: `refresh-token.error.insufficient_permissions`,
|
||||
workspaceId,
|
||||
connectedAccountId: messageChannel.connectedAccountId,
|
||||
messageChannelId: messageChannel.id,
|
||||
message: `${error.code}: ${error.reason}`,
|
||||
});
|
||||
|
||||
await this.messagingChannelSyncStatusService.markAsFailedInsufficientPermissionsAndFlushMessagesToImport(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
await this.connectedAccountRepository.updateAuthFailedAt(
|
||||
messageChannel.connectedAccountId,
|
||||
workspaceId,
|
||||
);
|
||||
}
|
||||
|
||||
@ -8,7 +8,6 @@ import {
|
||||
import { CONNECTED_ACCOUNT_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
|
||||
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
|
||||
import { CalendarChannelWorkspaceEntity } from 'src/modules/calendar/standard-objects/calendar-channel.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity';
|
||||
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
|
||||
import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator';
|
||||
@ -17,6 +16,7 @@ import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/work
|
||||
import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field.decorator';
|
||||
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
|
||||
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
|
||||
export enum ConnectedAccountProvider {
|
||||
GOOGLE = 'google',
|
||||
|
||||
@ -5,11 +5,11 @@ import { MessageQueueJob } from 'src/engine/integrations/message-queue/interface
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { BlocklistRepository } from 'src/modules/connected-account/repositories/blocklist.repository';
|
||||
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { ThreadCleanerService } from 'src/modules/messaging/services/thread-cleaner/thread-cleaner.service';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { MessagingMessageCleanerService } from 'src/modules/messaging/message-cleaner/services/messaging-message-cleaner.service';
|
||||
|
||||
export type BlocklistItemDeleteMessagesJobData = {
|
||||
workspaceId: string;
|
||||
@ -31,7 +31,7 @@ export class BlocklistItemDeleteMessagesJob
|
||||
private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository,
|
||||
@InjectObjectMetadataRepository(BlocklistWorkspaceEntity)
|
||||
private readonly blocklistRepository: BlocklistRepository,
|
||||
private readonly threadCleanerService: ThreadCleanerService,
|
||||
private readonly threadCleanerService: MessagingMessageCleanerService,
|
||||
) {}
|
||||
|
||||
async handle(data: BlocklistItemDeleteMessagesJobData): Promise<void> {
|
||||
@ -5,7 +5,6 @@ import { MessageQueueJob } from 'src/engine/integrations/message-queue/interface
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { GmailFullMessageListFetchService } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch.service';
|
||||
|
||||
export type BlocklistReimportMessagesJobData = {
|
||||
workspaceId: string;
|
||||
@ -22,7 +21,6 @@ export class BlocklistReimportMessagesJob
|
||||
constructor(
|
||||
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
|
||||
private readonly connectedAccountRepository: ConnectedAccountRepository,
|
||||
private readonly gmailFullMessageListFetchService: GmailFullMessageListFetchService,
|
||||
) {}
|
||||
|
||||
async handle(data: BlocklistReimportMessagesJobData): Promise<void> {
|
||||
@ -46,11 +44,12 @@ export class BlocklistReimportMessagesJob
|
||||
return;
|
||||
}
|
||||
|
||||
await this.gmailFullMessageListFetchService.fetchConnectedAccountThreads(
|
||||
workspaceId,
|
||||
connectedAccount[0].id,
|
||||
[handle],
|
||||
);
|
||||
// TODO: reimplement that
|
||||
// await this.gmailMessageListFetchJob.fetchConnectedAccountThreads(
|
||||
// workspaceId,
|
||||
// connectedAccount[0].id,
|
||||
// [handle],
|
||||
// );
|
||||
|
||||
this.logger.log(
|
||||
`Reimporting messages from ${handle} in workspace ${workspaceId} for workspace member ${workspaceMemberId} done`,
|
||||
@ -9,11 +9,11 @@ import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard
|
||||
import {
|
||||
BlocklistReimportMessagesJob,
|
||||
BlocklistReimportMessagesJobData,
|
||||
} from 'src/modules/messaging/jobs/blocklist-reimport-messages.job';
|
||||
} from 'src/modules/messaging/blocklist-manager/jobs/messaging-blocklist-reimport-messages.job';
|
||||
import {
|
||||
BlocklistItemDeleteMessagesJobData,
|
||||
BlocklistItemDeleteMessagesJob,
|
||||
} from 'src/modules/messaging/jobs/blocklist-item-delete-messages.job';
|
||||
} from 'src/modules/messaging/blocklist-manager/jobs/messaging-blocklist-item-delete-messages.job';
|
||||
import { ObjectRecordUpdateEvent } from 'src/engine/integrations/event-emitter/types/object-record-update.event';
|
||||
|
||||
@Injectable()
|
||||
@ -0,0 +1,8 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
|
||||
@Module({
|
||||
imports: [],
|
||||
providers: [],
|
||||
exports: [],
|
||||
})
|
||||
export class MessagingBlocklistManagerModule {}
|
||||
@ -0,0 +1,60 @@
|
||||
import { HttpModule } from '@nestjs/axios';
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { AnalyticsModule } from 'src/engine/core-modules/analytics/analytics.module';
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { WorkspaceDataSourceModule } from 'src/engine/workspace-datasource/workspace-datasource.module';
|
||||
import { AddPersonIdAndWorkspaceMemberIdModule } from 'src/modules/calendar-messaging-participant/services/add-person-id-and-workspace-member-id/add-person-id-and-workspace-member-id.module';
|
||||
import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
|
||||
import { MessagingErrorHandlingService } from 'src/modules/messaging/common/services/messaging-error-handling.service';
|
||||
import { MessagingFetchByBatchesService } from 'src/modules/messaging/common/services/messaging-fetch-by-batch.service';
|
||||
import { MessagingMessageParticipantService } from 'src/modules/messaging/common/services/messaging-message-participant.service';
|
||||
import { MessagingMessageThreadService } from 'src/modules/messaging/common/services/messaging-message-thread.service';
|
||||
import { MessagingMessageService } from 'src/modules/messaging/common/services/messaging-message.service';
|
||||
import { MessagingSaveMessagesAndEnqueueContactCreationService } from 'src/modules/messaging/common/services/messaging-save-messages-and-enqueue-contact-creation.service';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
|
||||
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-participant.workspace-entity';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
|
||||
import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
HttpModule.register({
|
||||
baseURL: 'https://www.googleapis.com/batch/gmail/v1',
|
||||
}),
|
||||
AnalyticsModule,
|
||||
WorkspaceDataSourceModule,
|
||||
ObjectMetadataRepositoryModule.forFeature([
|
||||
PersonWorkspaceEntity,
|
||||
MessageParticipantWorkspaceEntity,
|
||||
MessageWorkspaceEntity,
|
||||
MessageThreadWorkspaceEntity,
|
||||
]),
|
||||
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
|
||||
AddPersonIdAndWorkspaceMemberIdModule,
|
||||
],
|
||||
providers: [
|
||||
MessagingMessageService,
|
||||
MessagingMessageThreadService,
|
||||
MessagingSaveMessagesAndEnqueueContactCreationService,
|
||||
MessagingErrorHandlingService,
|
||||
MessagingTelemetryService,
|
||||
MessagingChannelSyncStatusService,
|
||||
MessagingMessageParticipantService,
|
||||
MessagingFetchByBatchesService,
|
||||
],
|
||||
exports: [
|
||||
MessagingMessageService,
|
||||
MessagingMessageThreadService,
|
||||
MessagingSaveMessagesAndEnqueueContactCreationService,
|
||||
MessagingErrorHandlingService,
|
||||
MessagingTelemetryService,
|
||||
MessagingChannelSyncStatusService,
|
||||
MessagingMessageParticipantService,
|
||||
MessagingFetchByBatchesService,
|
||||
],
|
||||
})
|
||||
export class MessagingCommonModule {}
|
||||
@ -10,15 +10,15 @@ import groupBy from 'lodash.groupby';
|
||||
import { WorkspacePreQueryHook } from 'src/engine/api/graphql/workspace-query-runner/workspace-pre-query-hook/interfaces/workspace-pre-query-hook.interface';
|
||||
import { FindManyResolverArgs } from 'src/engine/api/graphql/workspace-resolver-builder/interfaces/workspace-resolvers-builder.interface';
|
||||
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { WorkspaceMemberRepository } from 'src/modules/workspace-member/repositories/workspace-member.repository';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
|
||||
@Injectable()
|
||||
export class MessageFindManyPreQueryHook implements WorkspacePreQueryHook {
|
||||
@ -1,12 +1,12 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
|
||||
import { MessageFindManyPreQueryHook } from 'src/modules/messaging/query-hooks/message/message-find-many.pre-query.hook';
|
||||
import { MessageFindOnePreQueryHook } from 'src/modules/messaging/query-hooks/message/message-find-one.pre-query-hook';
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageFindManyPreQueryHook } from 'src/modules/messaging/common/query-hooks/message/message-find-many.pre-query.hook';
|
||||
import { MessageFindOnePreQueryHook } from 'src/modules/messaging/common/query-hooks/message/message-find-one.pre-query-hook';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
@ -3,8 +3,8 @@ import { Injectable } from '@nestjs/common';
|
||||
import { EntityManager } from 'typeorm';
|
||||
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
|
||||
@Injectable()
|
||||
export class MessageChannelMessageAssociationRepository {
|
||||
@ -3,12 +3,12 @@ import { Injectable } from '@nestjs/common';
|
||||
import { EntityManager } from 'typeorm';
|
||||
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import {
|
||||
MessageChannelWorkspaceEntity,
|
||||
MessageChannelSyncStatus,
|
||||
MessageChannelSyncSubStatus,
|
||||
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
} from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
|
||||
@Injectable()
|
||||
export class MessageChannelRepository {
|
||||
@ -3,9 +3,9 @@ import { Injectable } from '@nestjs/common';
|
||||
import { EntityManager } from 'typeorm';
|
||||
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-participant.workspace-entity';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { ParticipantWithId } from 'src/modules/messaging/types/gmail-message';
|
||||
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-participant.workspace-entity';
|
||||
import { ParticipantWithId } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
|
||||
|
||||
@Injectable()
|
||||
export class MessageParticipantRepository {
|
||||
@ -3,8 +3,8 @@ import { Injectable } from '@nestjs/common';
|
||||
import { EntityManager } from 'typeorm';
|
||||
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
|
||||
|
||||
@Injectable()
|
||||
export class MessageRepository {
|
||||
@ -4,15 +4,15 @@ import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache
|
||||
import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator';
|
||||
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import {
|
||||
MessageChannelWorkspaceEntity,
|
||||
MessageChannelSyncSubStatus,
|
||||
MessageChannelSyncStatus,
|
||||
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
} from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
|
||||
@Injectable()
|
||||
export class MessageChannelSyncStatusService {
|
||||
export class MessagingChannelSyncStatusService {
|
||||
constructor(
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
@ -6,9 +6,9 @@ import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repos
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelSyncStatusService } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.service';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/services/telemetry/messaging-telemetry.service';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
|
||||
|
||||
type SyncStep =
|
||||
| 'partial-message-list-fetch'
|
||||
@ -21,11 +21,11 @@ export type GmailError = {
|
||||
};
|
||||
|
||||
@Injectable()
|
||||
export class GmailErrorHandlingService {
|
||||
export class MessagingErrorHandlingService {
|
||||
constructor(
|
||||
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
|
||||
private readonly connectedAccountRepository: ConnectedAccountRepository,
|
||||
private readonly messageChannelSyncStatusService: MessageChannelSyncStatusService,
|
||||
private readonly messagingChannelSyncStatusService: MessagingChannelSyncStatusService,
|
||||
private readonly messagingTelemetryService: MessagingTelemetryService,
|
||||
) {}
|
||||
|
||||
@ -92,7 +92,7 @@ export class GmailErrorHandlingService {
|
||||
break;
|
||||
|
||||
default:
|
||||
await this.messageChannelSyncStatusService.markAsFailedUnknownAndFlushMessagesToImport(
|
||||
await this.messagingChannelSyncStatusService.markAsFailedUnknownAndFlushMessagesToImport(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -116,21 +116,21 @@ export class GmailErrorHandlingService {
|
||||
|
||||
switch (syncStep) {
|
||||
case 'full-message-list-fetch':
|
||||
await this.messageChannelSyncStatusService.scheduleFullMessageListFetch(
|
||||
await this.messagingChannelSyncStatusService.scheduleFullMessageListFetch(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
break;
|
||||
|
||||
case 'partial-message-list-fetch':
|
||||
await this.messageChannelSyncStatusService.schedulePartialMessageListFetch(
|
||||
await this.messagingChannelSyncStatusService.schedulePartialMessageListFetch(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
break;
|
||||
|
||||
case 'messages-import':
|
||||
await this.messageChannelSyncStatusService.scheduleMessagesImport(
|
||||
await this.messagingChannelSyncStatusService.scheduleMessagesImport(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -155,7 +155,7 @@ export class GmailErrorHandlingService {
|
||||
message: `${error.code}: ${error.reason}`,
|
||||
});
|
||||
|
||||
await this.messageChannelSyncStatusService.markAsFailedInsufficientPermissionsAndFlushMessagesToImport(
|
||||
await this.messagingChannelSyncStatusService.markAsFailedInsufficientPermissionsAndFlushMessagesToImport(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -184,7 +184,7 @@ export class GmailErrorHandlingService {
|
||||
message: `404: ${error.reason}`,
|
||||
});
|
||||
|
||||
await this.messageChannelSyncStatusService.resetAndScheduleFullMessageListFetch(
|
||||
await this.messagingChannelSyncStatusService.resetAndScheduleFullMessageListFetch(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -3,11 +3,11 @@ import { Injectable } from '@nestjs/common';
|
||||
|
||||
import { AxiosResponse } from 'axios';
|
||||
|
||||
import { BatchQueries } from 'src/modules/messaging/types/batch-queries';
|
||||
import { GmailMessageParsedResponse } from 'src/modules/messaging/types/gmail-message-parsed-response';
|
||||
import { GmailMessageParsedResponse } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message-parsed-response';
|
||||
import { BatchQueries } from 'src/modules/messaging/message-import-manager/types/batch-queries';
|
||||
|
||||
@Injectable()
|
||||
export class FetchByBatchesService {
|
||||
export class MessagingFetchByBatchesService {
|
||||
constructor(private readonly httpService: HttpService) {}
|
||||
|
||||
async fetchAllByBatches(
|
||||
@ -3,18 +3,20 @@ import { Injectable } from '@nestjs/common';
|
||||
import { EntityManager } from 'typeorm';
|
||||
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ParticipantWithMessageId } from 'src/modules/messaging/types/gmail-message';
|
||||
import { PersonRepository } from 'src/modules/person/repositories/person.repository';
|
||||
import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity';
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import { getFlattenedValuesAndValuesStringForBatchRawQuery } from 'src/modules/calendar/utils/get-flattened-values-and-values-string-for-batch-raw-query.util';
|
||||
import { MessageParticipantRepository } from 'src/modules/messaging/repositories/message-participant.repository';
|
||||
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-participant.workspace-entity';
|
||||
import { AddPersonIdAndWorkspaceMemberIdService } from 'src/modules/calendar-messaging-participant/services/add-person-id-and-workspace-member-id/add-person-id-and-workspace-member-id.service';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { MessageParticipantRepository } from 'src/modules/messaging/common/repositories/message-participant.repository';
|
||||
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-participant.workspace-entity';
|
||||
import { ParticipantWithMessageId } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
|
||||
|
||||
// Todo: this is not the right place for this file. The code needs to be refactored in term of business modules with a precise scope.
|
||||
// Putting it here to avoid circular dependencies for now.
|
||||
@Injectable()
|
||||
export class MessageParticipantService {
|
||||
export class MessagingMessageParticipantService {
|
||||
constructor(
|
||||
private readonly workspaceDataSourceService: WorkspaceDataSourceService,
|
||||
@InjectObjectMetadataRepository(MessageParticipantWorkspaceEntity)
|
||||
@ -4,15 +4,15 @@ import { EntityManager } from 'typeorm';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
|
||||
import { MessageRepository } from 'src/modules/messaging/repositories/message.repository';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity';
|
||||
import { MessageThreadRepository } from 'src/modules/messaging/repositories/message-thread.repository';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
|
||||
import { MessageThreadRepository } from 'src/modules/messaging/common/repositories/message-thread.repository';
|
||||
import { MessageRepository } from 'src/modules/messaging/common/repositories/message.repository';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
|
||||
|
||||
@Injectable()
|
||||
export class MessageThreadService {
|
||||
export class MessagingMessageThreadService {
|
||||
constructor(
|
||||
@InjectObjectMetadataRepository(
|
||||
MessageChannelMessageAssociationWorkspaceEntity,
|
||||
@ -4,23 +4,23 @@ import { DataSource, EntityManager } from 'typeorm';
|
||||
import { v4 } from 'uuid';
|
||||
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { GmailMessage } from 'src/modules/messaging/types/gmail-message';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
|
||||
import { MessageRepository } from 'src/modules/messaging/repositories/message.repository';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { MessageThreadService } from 'src/modules/messaging/services/message-thread/message-thread.service';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity';
|
||||
import { MessageThreadRepository } from 'src/modules/messaging/repositories/message-thread.repository';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import { MessageThreadRepository } from 'src/modules/messaging/common/repositories/message-thread.repository';
|
||||
import { MessageRepository } from 'src/modules/messaging/common/repositories/message.repository';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
|
||||
import { GmailMessage } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
|
||||
import { MessagingMessageThreadService } from 'src/modules/messaging/common/services/messaging-message-thread.service';
|
||||
|
||||
@Injectable()
|
||||
export class MessageService {
|
||||
private readonly logger = new Logger(MessageService.name);
|
||||
export class MessagingMessageService {
|
||||
private readonly logger = new Logger(MessagingMessageService.name);
|
||||
|
||||
constructor(
|
||||
private readonly workspaceDataSourceService: WorkspaceDataSourceService,
|
||||
@ -34,7 +34,7 @@ export class MessageService {
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
@InjectObjectMetadataRepository(MessageThreadWorkspaceEntity)
|
||||
private readonly messageThreadRepository: MessageThreadRepository,
|
||||
private readonly messageThreadService: MessageThreadService,
|
||||
private readonly messageThreadService: MessagingMessageThreadService,
|
||||
) {}
|
||||
|
||||
public async saveMessagesWithinTransaction(
|
||||
@ -8,13 +8,6 @@ import { MessageQueueService } from 'src/engine/integrations/message-queue/servi
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageParticipantService } from 'src/modules/messaging/services/message-participant/message-participant.service';
|
||||
import { MessageService } from 'src/modules/messaging/services/message/message.service';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import {
|
||||
GmailMessage,
|
||||
ParticipantWithMessageId,
|
||||
} from 'src/modules/messaging/types/gmail-message';
|
||||
import {
|
||||
CreateCompanyAndContactJobData,
|
||||
CreateCompanyAndContactJob,
|
||||
@ -23,15 +16,22 @@ import {
|
||||
FeatureFlagEntity,
|
||||
FeatureFlagKeys,
|
||||
} from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import {
|
||||
GmailMessage,
|
||||
ParticipantWithMessageId,
|
||||
} from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
|
||||
import { MessagingMessageService } from 'src/modules/messaging/common/services/messaging-message.service';
|
||||
import { MessagingMessageParticipantService } from 'src/modules/messaging/common/services/messaging-message-participant.service';
|
||||
|
||||
@Injectable()
|
||||
export class SaveMessagesAndEnqueueContactCreationService {
|
||||
export class MessagingSaveMessagesAndEnqueueContactCreationService {
|
||||
constructor(
|
||||
private readonly workspaceDataSourceService: WorkspaceDataSourceService,
|
||||
@Inject(MessageQueue.messagingQueue)
|
||||
private readonly messageQueueService: MessageQueueService,
|
||||
private readonly messageService: MessageService,
|
||||
private readonly messageParticipantService: MessageParticipantService,
|
||||
private readonly messageService: MessagingMessageService,
|
||||
private readonly messageParticipantService: MessagingMessageParticipantService,
|
||||
@InjectRepository(FeatureFlagEntity, 'core')
|
||||
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
|
||||
) {}
|
||||
@ -3,9 +3,6 @@ import { Relation } from 'src/engine/workspace-manager/workspace-sync-metadata/i
|
||||
import { FieldMetadataType } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
|
||||
import { MESSAGE_CHANNEL_MESSAGE_ASSOCIATION_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
|
||||
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
|
||||
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
|
||||
import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator';
|
||||
import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator';
|
||||
@ -14,6 +11,9 @@ import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field
|
||||
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
|
||||
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
|
||||
import { RelationMetadataType } from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
|
||||
|
||||
@WorkspaceEntity({
|
||||
standardId: STANDARD_OBJECT_IDS.messageChannelMessageAssociation,
|
||||
@ -8,7 +8,6 @@ import {
|
||||
import { MESSAGE_CHANNEL_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
|
||||
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
|
||||
import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator';
|
||||
import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator';
|
||||
@ -16,6 +15,7 @@ import { WorkspaceIsSystem } from 'src/engine/twenty-orm/decorators/workspace-is
|
||||
import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field.decorator';
|
||||
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
|
||||
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
|
||||
export enum MessageChannelSyncStatus {
|
||||
// TO BE DEPRECATED
|
||||
@ -3,7 +3,6 @@ import { Relation } from 'src/engine/workspace-manager/workspace-sync-metadata/i
|
||||
import { FieldMetadataType } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
|
||||
import { MESSAGE_PARTICIPANT_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
|
||||
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
|
||||
import { PersonWorkspaceEntity } from 'src/modules/person/standard-objects/person.workspace-entity';
|
||||
import { WorkspaceMemberWorkspaceEntity } from 'src/modules/workspace-member/standard-objects/workspace-member.workspace-entity';
|
||||
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
|
||||
@ -14,6 +13,7 @@ import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field
|
||||
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
|
||||
import { RelationMetadataType } from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity';
|
||||
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
|
||||
|
||||
@WorkspaceEntity({
|
||||
standardId: STANDARD_OBJECT_IDS.messageParticipant,
|
||||
@ -6,14 +6,14 @@ import {
|
||||
} from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity';
|
||||
import { MESSAGE_THREAD_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
|
||||
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
|
||||
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
|
||||
import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator';
|
||||
import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator';
|
||||
import { WorkspaceIsSystem } from 'src/engine/twenty-orm/decorators/workspace-is-system.decorator';
|
||||
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
|
||||
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
|
||||
|
||||
@WorkspaceEntity({
|
||||
standardId: STANDARD_OBJECT_IDS.messageThread,
|
||||
@ -7,9 +7,6 @@ import {
|
||||
} from 'src/engine/metadata-modules/relation-metadata/relation-metadata.entity';
|
||||
import { MESSAGE_STANDARD_FIELD_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids';
|
||||
import { STANDARD_OBJECT_IDS } from 'src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-participant.workspace-entity';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity';
|
||||
import { BaseWorkspaceEntity } from 'src/engine/twenty-orm/base.workspace-entity';
|
||||
import { WorkspaceEntity } from 'src/engine/twenty-orm/decorators/workspace-entity.decorator';
|
||||
import { WorkspaceIsNotAuditLogged } from 'src/engine/twenty-orm/decorators/workspace-is-not-audit-logged.decorator';
|
||||
@ -17,6 +14,9 @@ import { WorkspaceIsSystem } from 'src/engine/twenty-orm/decorators/workspace-is
|
||||
import { WorkspaceField } from 'src/engine/twenty-orm/decorators/workspace-field.decorator';
|
||||
import { WorkspaceIsNullable } from 'src/engine/twenty-orm/decorators/workspace-is-nullable.decorator';
|
||||
import { WorkspaceRelation } from 'src/engine/twenty-orm/decorators/workspace-relation.decorator';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-participant.workspace-entity';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
|
||||
|
||||
@WorkspaceEntity({
|
||||
standardId: STANDARD_OBJECT_IDS.message,
|
||||
@ -1 +0,0 @@
|
||||
export const GMAIL_EXCLUDED_CATEGORIES = ['promotions', 'social', 'forums'];
|
||||
@ -1 +0,0 @@
|
||||
export const GMAIL_ONGOING_SYNC_TIMEOUT = 1000 * 60 * 60; // 1 hour
|
||||
@ -1 +0,0 @@
|
||||
export const GMAIL_USERS_HISTORY_MAX_RESULT = 500;
|
||||
@ -1 +0,0 @@
|
||||
export const GMAIL_USERS_MESSAGES_GET_BATCH_SIZE = 20;
|
||||
@ -1 +0,0 @@
|
||||
export const GMAIL_USERS_MESSAGES_LIST_MAX_RESULT = 500;
|
||||
@ -1,15 +0,0 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { GmailMessagesImportCronCommand } from 'src/modules/messaging/crons/commands/gmail-messages-import.cron.command';
|
||||
import { GmailMessageListFetchCronCommand } from 'src/modules/messaging/crons/commands/gmail-message-list-fetch.cron.command';
|
||||
@Module({
|
||||
imports: [
|
||||
ObjectMetadataRepositoryModule.forFeature([
|
||||
ConnectedAccountWorkspaceEntity,
|
||||
]),
|
||||
],
|
||||
providers: [GmailMessageListFetchCronCommand, GmailMessagesImportCronCommand],
|
||||
})
|
||||
export class MessagingCronCommandsModule {}
|
||||
@ -1,117 +0,0 @@
|
||||
import { Injectable, Logger } from '@nestjs/common';
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { Repository, In } from 'typeorm';
|
||||
|
||||
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
|
||||
|
||||
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { GmailMessagesImportService } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import.service';
|
||||
import { EnvironmentService } from 'src/engine/integrations/environment/environment.service';
|
||||
import { GmailMessagesImportV2Service } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import-v2.service';
|
||||
import {
|
||||
FeatureFlagEntity,
|
||||
FeatureFlagKeys,
|
||||
} from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/services/telemetry/messaging-telemetry.service';
|
||||
|
||||
@Injectable()
|
||||
export class GmailMessagesImportCronJob implements MessageQueueJob<undefined> {
|
||||
private readonly logger = new Logger(GmailMessagesImportCronJob.name);
|
||||
|
||||
constructor(
|
||||
@InjectRepository(Workspace, 'core')
|
||||
private readonly workspaceRepository: Repository<Workspace>,
|
||||
@InjectRepository(DataSourceEntity, 'metadata')
|
||||
private readonly dataSourceRepository: Repository<DataSourceEntity>,
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
private readonly gmailFetchMessageContentFromCacheService: GmailMessagesImportService,
|
||||
private readonly gmailFetchMessageContentFromCacheV2Service: GmailMessagesImportV2Service,
|
||||
@InjectRepository(FeatureFlagEntity, 'core')
|
||||
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
|
||||
private readonly environmentService: EnvironmentService,
|
||||
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
|
||||
private readonly connectedAccountRepository: ConnectedAccountRepository,
|
||||
private readonly messagingTelemetryService: MessagingTelemetryService,
|
||||
) {}
|
||||
|
||||
async handle(): Promise<void> {
|
||||
const workspaceIds = (
|
||||
await this.workspaceRepository.find({
|
||||
where: this.environmentService.get('IS_BILLING_ENABLED')
|
||||
? {
|
||||
subscriptionStatus: In(['active', 'trialing', 'past_due']),
|
||||
}
|
||||
: {},
|
||||
select: ['id'],
|
||||
})
|
||||
).map((workspace) => workspace.id);
|
||||
|
||||
const dataSources = await this.dataSourceRepository.find({
|
||||
where: {
|
||||
workspaceId: In(workspaceIds),
|
||||
},
|
||||
});
|
||||
|
||||
const workspaceIdsWithDataSources = new Set(
|
||||
dataSources.map((dataSource) => dataSource.workspaceId),
|
||||
);
|
||||
|
||||
for (const workspaceId of workspaceIdsWithDataSources) {
|
||||
await this.fetchWorkspaceMessages(workspaceId);
|
||||
}
|
||||
}
|
||||
|
||||
private async fetchWorkspaceMessages(workspaceId: string): Promise<void> {
|
||||
const messageChannels =
|
||||
await this.messageChannelRepository.getAll(workspaceId);
|
||||
|
||||
const isGmailSyncV2EnabledFeatureFlag =
|
||||
await this.featureFlagRepository.findOneBy({
|
||||
workspaceId: workspaceId,
|
||||
key: FeatureFlagKeys.IsGmailSyncV2Enabled,
|
||||
value: true,
|
||||
});
|
||||
|
||||
const isGmailSyncV2Enabled = isGmailSyncV2EnabledFeatureFlag?.value;
|
||||
|
||||
for (const messageChannel of messageChannels) {
|
||||
if (!messageChannel?.isSyncEnabled) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (isGmailSyncV2Enabled) {
|
||||
await this.messagingTelemetryService.track({
|
||||
eventName: 'messages_import.triggered',
|
||||
workspaceId,
|
||||
connectedAccountId: messageChannel.connectedAccountId,
|
||||
messageChannelId: messageChannel.id,
|
||||
});
|
||||
|
||||
const connectedAccount =
|
||||
await this.connectedAccountRepository.getConnectedAccountOrThrow(
|
||||
workspaceId,
|
||||
messageChannel.connectedAccountId,
|
||||
);
|
||||
|
||||
await this.gmailFetchMessageContentFromCacheV2Service.processMessageBatchImport(
|
||||
messageChannel,
|
||||
connectedAccount,
|
||||
workspaceId,
|
||||
);
|
||||
} else {
|
||||
await this.gmailFetchMessageContentFromCacheService.fetchMessageContentFromCache(
|
||||
workspaceId,
|
||||
messageChannel.connectedAccountId,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,33 +0,0 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity';
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { GmailMessagesImportCronJob } from 'src/modules/messaging/crons/jobs/gmail-messages-import.cron.job';
|
||||
import { GmailMessageListFetchCronJob } from 'src/modules/messaging/crons/jobs/gmail-message-list-fetch.cron.job';
|
||||
import { GmailMessagesImportModule } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import.module';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessagingTelemetryModule } from 'src/modules/messaging/services/telemetry/messaging-telemetry.module';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
TypeOrmModule.forFeature([Workspace, FeatureFlagEntity], 'core'),
|
||||
TypeOrmModule.forFeature([DataSourceEntity], 'metadata'),
|
||||
ObjectMetadataRepositoryModule.forFeature([MessageChannelWorkspaceEntity]),
|
||||
GmailMessagesImportModule,
|
||||
MessagingTelemetryModule,
|
||||
],
|
||||
providers: [
|
||||
{
|
||||
provide: GmailMessagesImportCronJob.name,
|
||||
useClass: GmailMessagesImportCronJob,
|
||||
},
|
||||
{
|
||||
provide: GmailMessageListFetchCronJob.name,
|
||||
useClass: GmailMessageListFetchCronJob,
|
||||
},
|
||||
],
|
||||
})
|
||||
export class MessagingCronJobModule {}
|
||||
@ -1,126 +0,0 @@
|
||||
import { Injectable, Logger } from '@nestjs/common';
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { Repository } from 'typeorm';
|
||||
|
||||
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
|
||||
|
||||
import {
|
||||
FeatureFlagEntity,
|
||||
FeatureFlagKeys,
|
||||
} from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { GoogleAPIRefreshAccessTokenService } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.service';
|
||||
import { GmailFullMessageListFetchService } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch.service';
|
||||
import { GmailFullMessageListFetchV2Service } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch-v2.service';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/services/telemetry/messaging-telemetry.service';
|
||||
|
||||
export type GmailFullMessageListFetchJobData = {
|
||||
workspaceId: string;
|
||||
connectedAccountId: string;
|
||||
};
|
||||
|
||||
@Injectable()
|
||||
export class GmailFullMessageListFetchJob
|
||||
implements MessageQueueJob<GmailFullMessageListFetchJobData>
|
||||
{
|
||||
private readonly logger = new Logger(GmailFullMessageListFetchJob.name);
|
||||
|
||||
constructor(
|
||||
private readonly googleAPIsRefreshAccessTokenService: GoogleAPIRefreshAccessTokenService,
|
||||
private readonly gmailFullMessageListFetchService: GmailFullMessageListFetchService,
|
||||
@InjectRepository(FeatureFlagEntity, 'core')
|
||||
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
|
||||
private readonly gmailFullMessageListFetchV2Service: GmailFullMessageListFetchV2Service,
|
||||
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
|
||||
private readonly connectedAccountRepository: ConnectedAccountRepository,
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
private readonly messagingTelemetryService: MessagingTelemetryService,
|
||||
) {}
|
||||
|
||||
async handle(data: GmailFullMessageListFetchJobData): Promise<void> {
|
||||
const { workspaceId, connectedAccountId } = data;
|
||||
|
||||
this.logger.log(
|
||||
`gmail full-sync for workspace ${workspaceId} and account ${connectedAccountId}`,
|
||||
);
|
||||
|
||||
try {
|
||||
await this.googleAPIsRefreshAccessTokenService.refreshAndSaveAccessToken(
|
||||
workspaceId,
|
||||
connectedAccountId,
|
||||
);
|
||||
} catch (e) {
|
||||
this.logger.error(
|
||||
`Error refreshing access token for connected account ${connectedAccountId} in workspace ${workspaceId}`,
|
||||
e,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
const isGmailSyncV2EnabledFeatureFlag =
|
||||
await this.featureFlagRepository.findOneBy({
|
||||
workspaceId: workspaceId,
|
||||
key: FeatureFlagKeys.IsGmailSyncV2Enabled,
|
||||
value: true,
|
||||
});
|
||||
|
||||
const isGmailSyncV2Enabled = isGmailSyncV2EnabledFeatureFlag?.value;
|
||||
|
||||
if (isGmailSyncV2Enabled) {
|
||||
// Todo delete this code block after migration
|
||||
const connectedAccount = await this.connectedAccountRepository.getById(
|
||||
connectedAccountId,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
if (!connectedAccount) {
|
||||
throw new Error(
|
||||
`Connected account ${connectedAccountId} not found in workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
|
||||
const messageChannel =
|
||||
await this.messageChannelRepository.getFirstByConnectedAccountId(
|
||||
connectedAccountId,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
if (!messageChannel) {
|
||||
throw new Error(
|
||||
`No message channel found for connected account ${connectedAccountId} in workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
|
||||
await this.messagingTelemetryService.track({
|
||||
eventName: 'full_message_list_fetch.started',
|
||||
workspaceId,
|
||||
connectedAccountId,
|
||||
});
|
||||
|
||||
await this.gmailFullMessageListFetchV2Service.processMessageListFetch(
|
||||
messageChannel,
|
||||
connectedAccount,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
await this.messagingTelemetryService.track({
|
||||
eventName: 'full_message_list_fetch.completed',
|
||||
workspaceId,
|
||||
connectedAccountId,
|
||||
messageChannelId: messageChannel.id,
|
||||
});
|
||||
} else {
|
||||
await this.gmailFullMessageListFetchService.fetchConnectedAccountThreads(
|
||||
data.workspaceId,
|
||||
data.connectedAccountId,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,48 +0,0 @@
|
||||
import { Injectable, Logger } from '@nestjs/common';
|
||||
|
||||
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
|
||||
|
||||
import { GoogleAPIRefreshAccessTokenService } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.service';
|
||||
import { GmailPartialMessageListFetchService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-partial-message-list-fetch.service';
|
||||
|
||||
export type GmailPartialMessageListFetchJobData = {
|
||||
workspaceId: string;
|
||||
connectedAccountId: string;
|
||||
};
|
||||
|
||||
@Injectable()
|
||||
export class GmailPartialMessageListFetchJob
|
||||
implements MessageQueueJob<GmailPartialMessageListFetchJobData>
|
||||
{
|
||||
private readonly logger = new Logger(GmailPartialMessageListFetchJob.name);
|
||||
|
||||
constructor(
|
||||
private readonly googleAPIsRefreshAccessTokenService: GoogleAPIRefreshAccessTokenService,
|
||||
private readonly gmailPartialSyncService: GmailPartialMessageListFetchService,
|
||||
) {}
|
||||
|
||||
async handle(data: GmailPartialMessageListFetchJobData): Promise<void> {
|
||||
this.logger.log(
|
||||
`gmail partial-sync for workspace ${data.workspaceId} and account ${data.connectedAccountId}`,
|
||||
);
|
||||
|
||||
try {
|
||||
await this.googleAPIsRefreshAccessTokenService.refreshAndSaveAccessToken(
|
||||
data.workspaceId,
|
||||
data.connectedAccountId,
|
||||
);
|
||||
} catch (e) {
|
||||
this.logger.error(
|
||||
`Error refreshing access token for connected account ${data.connectedAccountId} in workspace ${data.workspaceId}`,
|
||||
e,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
await this.gmailPartialSyncService.fetchConnectedAccountThreads(
|
||||
data.workspaceId,
|
||||
data.connectedAccountId,
|
||||
);
|
||||
}
|
||||
}
|
||||
@ -1,75 +0,0 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { AutoCompaniesAndContactsCreationModule } from 'src/modules/connected-account/auto-companies-and-contacts-creation/auto-companies-and-contacts-creation.module';
|
||||
import { GoogleAPIRefreshAccessTokenModule } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.module';
|
||||
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { BlocklistItemDeleteMessagesJob } from 'src/modules/messaging/jobs/blocklist-item-delete-messages.job';
|
||||
import { BlocklistReimportMessagesJob } from 'src/modules/messaging/jobs/blocklist-reimport-messages.job';
|
||||
import { DeleteConnectedAccountAssociatedMessagingDataJob } from 'src/modules/messaging/jobs/delete-connected-account-associated-messaging-data.job';
|
||||
import { GmailFullMessageListFetchJob } from 'src/modules/messaging/jobs/gmail-full-message-list-fetch.job';
|
||||
import { GmailMessageListFetchJob } from 'src/modules/messaging/jobs/gmail-message-list-fetch.job';
|
||||
import { GmailPartialMessageListFetchJob } from 'src/modules/messaging/jobs/gmail-partial-message-list-fetch.job';
|
||||
import { MessagingCreateCompanyAndContactAfterSyncJob } from 'src/modules/messaging/jobs/messaging-create-company-and-contact-after-sync.job';
|
||||
import { GmailFullMessageListFetchModule } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch.module';
|
||||
import { GmailPartialMessageListFetchModule } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-partial-message-list-fetch.module';
|
||||
import { SetMessageChannelSyncStatusModule } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.module';
|
||||
import { MessagingTelemetryModule } from 'src/modules/messaging/services/telemetry/messaging-telemetry.module';
|
||||
import { ThreadCleanerModule } from 'src/modules/messaging/services/thread-cleaner/thread-cleaner.module';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-participant.workspace-entity';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
ObjectMetadataRepositoryModule.forFeature([
|
||||
ConnectedAccountWorkspaceEntity,
|
||||
MessageChannelWorkspaceEntity,
|
||||
MessageParticipantWorkspaceEntity,
|
||||
MessageChannelMessageAssociationWorkspaceEntity,
|
||||
BlocklistWorkspaceEntity,
|
||||
]),
|
||||
MessagingTelemetryModule,
|
||||
GmailFullMessageListFetchModule,
|
||||
GmailPartialMessageListFetchModule,
|
||||
ThreadCleanerModule,
|
||||
GoogleAPIRefreshAccessTokenModule,
|
||||
AutoCompaniesAndContactsCreationModule,
|
||||
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
|
||||
SetMessageChannelSyncStatusModule,
|
||||
],
|
||||
providers: [
|
||||
{
|
||||
provide: BlocklistReimportMessagesJob.name,
|
||||
useClass: BlocklistReimportMessagesJob,
|
||||
},
|
||||
{
|
||||
provide: BlocklistItemDeleteMessagesJob.name,
|
||||
useClass: BlocklistItemDeleteMessagesJob,
|
||||
},
|
||||
{
|
||||
provide: GmailFullMessageListFetchJob.name,
|
||||
useClass: GmailFullMessageListFetchJob,
|
||||
},
|
||||
{
|
||||
provide: GmailPartialMessageListFetchJob.name,
|
||||
useClass: GmailPartialMessageListFetchJob,
|
||||
},
|
||||
{
|
||||
provide: GmailMessageListFetchJob.name,
|
||||
useClass: GmailMessageListFetchJob,
|
||||
},
|
||||
{
|
||||
provide: DeleteConnectedAccountAssociatedMessagingDataJob.name,
|
||||
useClass: DeleteConnectedAccountAssociatedMessagingDataJob,
|
||||
},
|
||||
{
|
||||
provide: MessagingCreateCompanyAndContactAfterSyncJob.name,
|
||||
useClass: MessagingCreateCompanyAndContactAfterSyncJob,
|
||||
},
|
||||
],
|
||||
})
|
||||
export class MessagingJobModule {}
|
||||
@ -2,32 +2,33 @@ import { Injectable, Logger } from '@nestjs/common';
|
||||
|
||||
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
|
||||
|
||||
import { ThreadCleanerService } from 'src/modules/messaging/services/thread-cleaner/thread-cleaner.service';
|
||||
import { MessagingMessageCleanerService } from 'src/modules/messaging/message-cleaner/services/messaging-message-cleaner.service';
|
||||
|
||||
export type DeleteConnectedAccountAssociatedMessagingDataJobData = {
|
||||
export type MessagingConnectedAccountDeletionCleanupJobData = {
|
||||
workspaceId: string;
|
||||
connectedAccountId: string;
|
||||
};
|
||||
|
||||
@Injectable()
|
||||
export class DeleteConnectedAccountAssociatedMessagingDataJob
|
||||
implements
|
||||
MessageQueueJob<DeleteConnectedAccountAssociatedMessagingDataJobData>
|
||||
export class MessagingConnectedAccountDeletionCleanupJob
|
||||
implements MessageQueueJob<MessagingConnectedAccountDeletionCleanupJobData>
|
||||
{
|
||||
private readonly logger = new Logger(
|
||||
DeleteConnectedAccountAssociatedMessagingDataJob.name,
|
||||
MessagingConnectedAccountDeletionCleanupJob.name,
|
||||
);
|
||||
|
||||
constructor(private readonly threadCleanerService: ThreadCleanerService) {}
|
||||
constructor(
|
||||
private readonly messageCleanerService: MessagingMessageCleanerService,
|
||||
) {}
|
||||
|
||||
async handle(
|
||||
data: DeleteConnectedAccountAssociatedMessagingDataJobData,
|
||||
data: MessagingConnectedAccountDeletionCleanupJobData,
|
||||
): Promise<void> {
|
||||
this.logger.log(
|
||||
`Deleting connected account ${data.connectedAccountId} associated messaging data in workspace ${data.workspaceId}`,
|
||||
);
|
||||
|
||||
await this.threadCleanerService.cleanWorkspaceThreads(data.workspaceId);
|
||||
await this.messageCleanerService.cleanWorkspaceThreads(data.workspaceId);
|
||||
|
||||
this.logger.log(
|
||||
`Deleted connected account ${data.connectedAccountId} associated messaging data in workspace ${data.workspaceId}`,
|
||||
@ -1,8 +1,5 @@
|
||||
import { Injectable, Inject } from '@nestjs/common';
|
||||
import { Inject } from '@nestjs/common';
|
||||
import { OnEvent } from '@nestjs/event-emitter';
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { Repository } from 'typeorm';
|
||||
|
||||
import { ObjectRecordDeleteEvent } from 'src/engine/integrations/event-emitter/types/object-record-delete.event';
|
||||
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
|
||||
@ -11,30 +8,26 @@ import {
|
||||
DeleteConnectedAccountAssociatedCalendarDataJobData,
|
||||
DeleteConnectedAccountAssociatedCalendarDataJob,
|
||||
} from 'src/modules/calendar/jobs/delete-connected-account-associated-calendar-data.job';
|
||||
import {
|
||||
DeleteConnectedAccountAssociatedMessagingDataJobData,
|
||||
DeleteConnectedAccountAssociatedMessagingDataJob,
|
||||
} from 'src/modules/messaging/jobs/delete-connected-account-associated-messaging-data.job';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import {
|
||||
MessagingConnectedAccountDeletionCleanupJob,
|
||||
MessagingConnectedAccountDeletionCleanupJobData,
|
||||
} from 'src/modules/messaging/message-cleaner/jobs/messaging-connected-account-deletion-cleanup.job';
|
||||
|
||||
@Injectable()
|
||||
export class MessagingConnectedAccountListener {
|
||||
export class MessagingMessageCleanerConnectedAccountListener {
|
||||
constructor(
|
||||
@Inject(MessageQueue.messagingQueue)
|
||||
private readonly messageQueueService: MessageQueueService,
|
||||
@Inject(MessageQueue.calendarQueue)
|
||||
private readonly calendarQueueService: MessageQueueService,
|
||||
@InjectRepository(FeatureFlagEntity, 'core')
|
||||
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
|
||||
) {}
|
||||
|
||||
@OnEvent('connectedAccount.deleted')
|
||||
async handleDeletedEvent(
|
||||
payload: ObjectRecordDeleteEvent<ConnectedAccountWorkspaceEntity>,
|
||||
) {
|
||||
await this.messageQueueService.add<DeleteConnectedAccountAssociatedMessagingDataJobData>(
|
||||
DeleteConnectedAccountAssociatedMessagingDataJob.name,
|
||||
await this.messageQueueService.add<MessagingConnectedAccountDeletionCleanupJobData>(
|
||||
MessagingConnectedAccountDeletionCleanupJob.name,
|
||||
{
|
||||
workspaceId: payload.workspaceId,
|
||||
connectedAccountId: payload.recordId,
|
||||
@ -0,0 +1,27 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
|
||||
import { MessagingConnectedAccountDeletionCleanupJob } from 'src/modules/messaging/message-cleaner/jobs/messaging-connected-account-deletion-cleanup.job';
|
||||
import { MessagingMessageCleanerConnectedAccountListener } from 'src/modules/messaging/message-cleaner/listeners/messaging-message-cleaner-connected-account.listener';
|
||||
import { MessagingMessageCleanerService } from 'src/modules/messaging/message-cleaner/services/messaging-message-cleaner.service';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
ObjectMetadataRepositoryModule.forFeature([
|
||||
MessageWorkspaceEntity,
|
||||
MessageThreadWorkspaceEntity,
|
||||
]),
|
||||
],
|
||||
providers: [
|
||||
MessagingMessageCleanerService,
|
||||
{
|
||||
provide: MessagingConnectedAccountDeletionCleanupJob.name,
|
||||
useClass: MessagingConnectedAccountDeletionCleanupJob,
|
||||
},
|
||||
MessagingMessageCleanerConnectedAccountListener,
|
||||
],
|
||||
exports: [MessagingMessageCleanerService],
|
||||
})
|
||||
export class MessagingMessageCleanerModule {}
|
||||
@ -1,14 +1,14 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { MessageThreadRepository } from 'src/modules/messaging/repositories/message-thread.repository';
|
||||
import { MessageRepository } from 'src/modules/messaging/repositories/message.repository';
|
||||
import { deleteUsingPagination } from 'src/modules/messaging/services/thread-cleaner/utils/delete-using-pagination.util';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-thread.workspace-entity';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/standard-objects/message.workspace-entity';
|
||||
import { MessageThreadRepository } from 'src/modules/messaging/common/repositories/message-thread.repository';
|
||||
import { MessageRepository } from 'src/modules/messaging/common/repositories/message.repository';
|
||||
import { MessageThreadWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-thread.workspace-entity';
|
||||
import { MessageWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message.workspace-entity';
|
||||
import { deleteUsingPagination } from 'src/modules/messaging/message-cleaner/utils/delete-using-pagination.util';
|
||||
|
||||
@Injectable()
|
||||
export class ThreadCleanerService {
|
||||
export class MessagingMessageCleanerService {
|
||||
constructor(
|
||||
@InjectObjectMetadataRepository(MessageWorkspaceEntity)
|
||||
private readonly messageRepository: MessageRepository,
|
||||
@ -1,4 +1,4 @@
|
||||
import { deleteUsingPagination } from './delete-using-pagination.util';
|
||||
import { deleteUsingPagination } from 'src/modules/messaging/message-cleaner/utils/delete-using-pagination.util';
|
||||
|
||||
describe('deleteUsingPagination', () => {
|
||||
it('should delete items using pagination', async () => {
|
||||
@ -29,7 +29,7 @@ describe('deleteUsingPagination', () => {
|
||||
expect(getterPaginated).toHaveBeenNthCalledWith(
|
||||
2,
|
||||
batchSize,
|
||||
batchSize,
|
||||
0,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
@ -0,0 +1 @@
|
||||
export const MESSAGING_IMPORT_ONGOING_SYNC_TIMEOUT = 1000 * 60 * 60; // 1 hour
|
||||
@ -4,16 +4,16 @@ import { Command, CommandRunner } from 'nest-commander';
|
||||
|
||||
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
|
||||
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
|
||||
import { GmailMessageListFetchCronJob } from 'src/modules/messaging/crons/jobs/gmail-message-list-fetch.cron.job';
|
||||
import { MessagingMessageListFetchCronJob } from 'src/modules/messaging/message-import-manager/crons/jobs/messaging-message-list-fetch.cron.job';
|
||||
|
||||
const GMAIL_MESSAGE_LIST_FETCH_CRON_PATTERN = '*/5 * * * *';
|
||||
const MESSAGING_MESSAGE_LIST_FETCH_CRON_PATTERN = '*/5 * * * *';
|
||||
|
||||
@Command({
|
||||
name: 'cron:messaging:gmail-message-list-fetch',
|
||||
name: 'cron:messaging:message-list-fetch',
|
||||
description:
|
||||
'Starts a cron job to sync existing connected account messages and store them in the cache',
|
||||
})
|
||||
export class GmailMessageListFetchCronCommand extends CommandRunner {
|
||||
export class MessagingMessageListFetchCronCommand extends CommandRunner {
|
||||
constructor(
|
||||
@Inject(MessageQueue.cronQueue)
|
||||
private readonly messageQueueService: MessageQueueService,
|
||||
@ -23,10 +23,10 @@ export class GmailMessageListFetchCronCommand extends CommandRunner {
|
||||
|
||||
async run(): Promise<void> {
|
||||
await this.messageQueueService.addCron<undefined>(
|
||||
GmailMessageListFetchCronJob.name,
|
||||
MessagingMessageListFetchCronJob.name,
|
||||
undefined,
|
||||
{
|
||||
repeat: { pattern: GMAIL_MESSAGE_LIST_FETCH_CRON_PATTERN },
|
||||
repeat: { pattern: MESSAGING_MESSAGE_LIST_FETCH_CRON_PATTERN },
|
||||
},
|
||||
);
|
||||
}
|
||||
@ -4,13 +4,13 @@ import { Command, CommandRunner } from 'nest-commander';
|
||||
|
||||
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
|
||||
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
|
||||
import { GmailMessagesImportCronJob } from 'src/modules/messaging/crons/jobs/gmail-messages-import.cron.job';
|
||||
import { MessagingMessagesImportCronJob } from 'src/modules/messaging/message-import-manager/crons/jobs/messaging-messages-import.cron.job';
|
||||
|
||||
@Command({
|
||||
name: 'cron:messaging:gmail-messages-import',
|
||||
name: 'cron:messaging:messages-import',
|
||||
description: 'Starts a cron job to fetch all messages from cache',
|
||||
})
|
||||
export class GmailMessagesImportCronCommand extends CommandRunner {
|
||||
export class MessagingMessagesImportCronCommand extends CommandRunner {
|
||||
constructor(
|
||||
@Inject(MessageQueue.cronQueue)
|
||||
private readonly messageQueueService: MessageQueueService,
|
||||
@ -20,7 +20,7 @@ export class GmailMessagesImportCronCommand extends CommandRunner {
|
||||
|
||||
async run(): Promise<void> {
|
||||
await this.messageQueueService.addCron<undefined>(
|
||||
GmailMessagesImportCronJob.name,
|
||||
MessagingMessagesImportCronJob.name,
|
||||
undefined,
|
||||
{
|
||||
repeat: {
|
||||
@ -7,30 +7,22 @@ import { MessageQueueJob } from 'src/engine/integrations/message-queue/interface
|
||||
|
||||
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity';
|
||||
import {
|
||||
GmailPartialMessageListFetchJobData,
|
||||
GmailPartialMessageListFetchJob,
|
||||
} from 'src/modules/messaging/jobs/gmail-partial-message-list-fetch.job';
|
||||
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
|
||||
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { EnvironmentService } from 'src/engine/integrations/environment/environment.service';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import {
|
||||
FeatureFlagEntity,
|
||||
FeatureFlagKeys,
|
||||
} from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import {
|
||||
GmailMessageListFetchJobData,
|
||||
GmailMessageListFetchJob,
|
||||
} from 'src/modules/messaging/jobs/gmail-message-list-fetch.job';
|
||||
MessagingMessageListFetchJobData,
|
||||
MessagingMessageListFetchJob,
|
||||
} from 'src/modules/messaging/message-import-manager/jobs/messaging-message-list-fetch.job';
|
||||
|
||||
@Injectable()
|
||||
export class GmailMessageListFetchCronJob
|
||||
export class MessagingMessageListFetchCronJob
|
||||
implements MessageQueueJob<undefined>
|
||||
{
|
||||
private readonly logger = new Logger(GmailMessageListFetchCronJob.name);
|
||||
private readonly logger = new Logger(MessagingMessageListFetchCronJob.name);
|
||||
|
||||
constructor(
|
||||
@InjectRepository(Workspace, 'core')
|
||||
@ -42,8 +34,6 @@ export class GmailMessageListFetchCronJob
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
private readonly environmentService: EnvironmentService,
|
||||
@InjectRepository(FeatureFlagEntity, 'core')
|
||||
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
|
||||
) {}
|
||||
|
||||
async handle(): Promise<void> {
|
||||
@ -74,15 +64,6 @@ export class GmailMessageListFetchCronJob
|
||||
}
|
||||
|
||||
private async enqueueSyncs(workspaceId: string): Promise<void> {
|
||||
const isGmailSyncV2EnabledFeatureFlag =
|
||||
await this.featureFlagRepository.findOneBy({
|
||||
workspaceId: workspaceId,
|
||||
key: FeatureFlagKeys.IsGmailSyncV2Enabled,
|
||||
value: true,
|
||||
});
|
||||
|
||||
const isGmailSyncV2Enabled = isGmailSyncV2EnabledFeatureFlag?.value;
|
||||
|
||||
try {
|
||||
const messageChannels =
|
||||
await this.messageChannelRepository.getAll(workspaceId);
|
||||
@ -92,23 +73,13 @@ export class GmailMessageListFetchCronJob
|
||||
continue;
|
||||
}
|
||||
|
||||
if (isGmailSyncV2Enabled) {
|
||||
await this.messageQueueService.add<GmailMessageListFetchJobData>(
|
||||
GmailMessageListFetchJob.name,
|
||||
{
|
||||
workspaceId,
|
||||
connectedAccountId: messageChannel.connectedAccountId,
|
||||
},
|
||||
);
|
||||
} else {
|
||||
await this.messageQueueService.add<GmailPartialMessageListFetchJobData>(
|
||||
GmailPartialMessageListFetchJob.name,
|
||||
{
|
||||
workspaceId,
|
||||
connectedAccountId: messageChannel.connectedAccountId,
|
||||
},
|
||||
);
|
||||
}
|
||||
await this.messageQueueService.add<MessagingMessageListFetchJobData>(
|
||||
MessagingMessageListFetchJob.name,
|
||||
{
|
||||
workspaceId,
|
||||
connectedAccountId: messageChannel.connectedAccountId,
|
||||
},
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
this.logger.error(
|
||||
@ -0,0 +1,65 @@
|
||||
import { Inject, Injectable, Logger } from '@nestjs/common';
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { Repository, In } from 'typeorm';
|
||||
|
||||
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
|
||||
|
||||
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity';
|
||||
import { EnvironmentService } from 'src/engine/integrations/environment/environment.service';
|
||||
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
|
||||
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
|
||||
import {
|
||||
MessagingMessagesImportJobData,
|
||||
MessagingMessagesImportJob,
|
||||
} from 'src/modules/messaging/message-import-manager/jobs/messaging-messages-import.job';
|
||||
|
||||
@Injectable()
|
||||
export class MessagingMessagesImportCronJob
|
||||
implements MessageQueueJob<undefined>
|
||||
{
|
||||
private readonly logger = new Logger(MessagingMessagesImportCronJob.name);
|
||||
|
||||
constructor(
|
||||
@InjectRepository(Workspace, 'core')
|
||||
private readonly workspaceRepository: Repository<Workspace>,
|
||||
@InjectRepository(DataSourceEntity, 'metadata')
|
||||
private readonly dataSourceRepository: Repository<DataSourceEntity>,
|
||||
private readonly environmentService: EnvironmentService,
|
||||
@Inject(MessageQueue.messagingQueue)
|
||||
private readonly messageQueueService: MessageQueueService,
|
||||
) {}
|
||||
|
||||
async handle(): Promise<void> {
|
||||
const workspaceIds = (
|
||||
await this.workspaceRepository.find({
|
||||
where: this.environmentService.get('IS_BILLING_ENABLED')
|
||||
? {
|
||||
subscriptionStatus: In(['active', 'trialing', 'past_due']),
|
||||
}
|
||||
: {},
|
||||
select: ['id'],
|
||||
})
|
||||
).map((workspace) => workspace.id);
|
||||
|
||||
const dataSources = await this.dataSourceRepository.find({
|
||||
where: {
|
||||
workspaceId: In(workspaceIds),
|
||||
},
|
||||
});
|
||||
|
||||
const workspaceIdsWithDataSources = new Set(
|
||||
dataSources.map((dataSource) => dataSource.workspaceId),
|
||||
);
|
||||
|
||||
for (const workspaceId of workspaceIdsWithDataSources) {
|
||||
await this.messageQueueService.add<MessagingMessagesImportJobData>(
|
||||
MessagingMessagesImportJob.name,
|
||||
{
|
||||
workspaceId,
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,5 @@
|
||||
export const MESSAGING_GMAIL_EXCLUDED_CATEGORIES = [
|
||||
'promotions',
|
||||
'social',
|
||||
'forums',
|
||||
];
|
||||
@ -0,0 +1 @@
|
||||
export const MESSAGING_GMAIL_USERS_HISTORY_MAX_RESULT = 500;
|
||||
@ -0,0 +1 @@
|
||||
export const MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE = 20;
|
||||
@ -0,0 +1 @@
|
||||
export const MESSAGING_GMAIL_USERS_MESSAGES_LIST_MAX_RESULT = 500;
|
||||
@ -0,0 +1,53 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { EnvironmentModule } from 'src/engine/integrations/environment/environment.module';
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { GoogleAPIRefreshAccessTokenModule } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.module';
|
||||
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { MessagingGmailClientProvider } from 'src/modules/messaging/message-import-manager/drivers/gmail/providers/messaging-gmail-client.provider';
|
||||
import { MessagingGmailFetchMessagesByBatchesService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-fetch-messages-by-batches.service';
|
||||
import { MessagingGmailFetchMessageIdsToExcludeService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-fetch-messages-ids-to-exclude.service';
|
||||
import { MessagingGmailFullMessageListFetchService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-full-message-list-fetch.service';
|
||||
import { MessagingGmailHistoryService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-history.service';
|
||||
import { MessagingGmailMessagesImportService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-messages-import.service';
|
||||
import { MessagingGmailPartialMessageListFetchService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-partial-message-list-fetch.service';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
GoogleAPIRefreshAccessTokenModule,
|
||||
EnvironmentModule,
|
||||
ObjectMetadataRepositoryModule.forFeature([
|
||||
ConnectedAccountWorkspaceEntity,
|
||||
MessageChannelWorkspaceEntity,
|
||||
MessageChannelMessageAssociationWorkspaceEntity,
|
||||
BlocklistWorkspaceEntity,
|
||||
]),
|
||||
MessagingCommonModule,
|
||||
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
|
||||
],
|
||||
providers: [
|
||||
MessagingGmailClientProvider,
|
||||
MessagingGmailHistoryService,
|
||||
MessagingGmailFetchMessagesByBatchesService,
|
||||
MessagingGmailPartialMessageListFetchService,
|
||||
MessagingGmailFullMessageListFetchService,
|
||||
MessagingGmailMessagesImportService,
|
||||
MessagingGmailFetchMessageIdsToExcludeService,
|
||||
],
|
||||
exports: [
|
||||
MessagingGmailClientProvider,
|
||||
MessagingGmailHistoryService,
|
||||
MessagingGmailFetchMessagesByBatchesService,
|
||||
MessagingGmailPartialMessageListFetchService,
|
||||
MessagingGmailFullMessageListFetchService,
|
||||
MessagingGmailMessagesImportService,
|
||||
MessagingGmailFetchMessageIdsToExcludeService,
|
||||
],
|
||||
})
|
||||
export class MessagingGmailDriverModule {}
|
||||
@ -6,7 +6,7 @@ import { gmail_v1, google } from 'googleapis';
|
||||
import { EnvironmentService } from 'src/engine/integrations/environment/environment.service';
|
||||
|
||||
@Injectable()
|
||||
export class GmailClientProvider {
|
||||
export class MessagingGmailClientProvider {
|
||||
constructor(private readonly environmentService: EnvironmentService) {}
|
||||
|
||||
public async getGmailClient(refreshToken: string): Promise<gmail_v1.Gmail> {
|
||||
@ -5,17 +5,21 @@ import planer from 'planer';
|
||||
import addressparser from 'addressparser';
|
||||
import { gmail_v1 } from 'googleapis';
|
||||
|
||||
import { GmailMessage } from 'src/modules/messaging/types/gmail-message';
|
||||
import { MessageQuery } from 'src/modules/messaging/types/message-or-thread-query';
|
||||
import { FetchByBatchesService } from 'src/modules/messaging/services/fetch-by-batch/fetch-by-batch.service';
|
||||
import { formatAddressObjectAsParticipants } from 'src/modules/messaging/services/utils/format-address-object-as-participants.util';
|
||||
import { assert, assertNotNull } from 'src/utils/assert';
|
||||
import { GmailMessage } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
|
||||
import { MessageQuery } from 'src/modules/messaging/message-import-manager/types/message-or-thread-query';
|
||||
import { formatAddressObjectAsParticipants } from 'src/modules/messaging/message-import-manager/utils/format-address-object-as-participants.util';
|
||||
import { MessagingFetchByBatchesService } from 'src/modules/messaging/common/services/messaging-fetch-by-batch.service';
|
||||
|
||||
@Injectable()
|
||||
export class FetchMessagesByBatchesService {
|
||||
private readonly logger = new Logger(FetchMessagesByBatchesService.name);
|
||||
export class MessagingGmailFetchMessagesByBatchesService {
|
||||
private readonly logger = new Logger(
|
||||
MessagingGmailFetchMessagesByBatchesService.name,
|
||||
);
|
||||
|
||||
constructor(private readonly fetchByBatchesService: FetchByBatchesService) {}
|
||||
constructor(
|
||||
private readonly fetchByBatchesService: MessagingFetchByBatchesService,
|
||||
) {}
|
||||
|
||||
async fetchAllMessages(
|
||||
queries: MessageQuery[],
|
||||
@ -2,15 +2,15 @@ import { Injectable } from '@nestjs/common';
|
||||
|
||||
import { gmail_v1 } from 'googleapis';
|
||||
|
||||
import { GMAIL_EXCLUDED_CATEGORIES } from 'src/modules/messaging/constants/gmail-excluded-categories';
|
||||
import { GmailGetHistoryService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-get-history.service';
|
||||
import { computeGmailCategoryLabelId } from 'src/modules/messaging/utils/compute-gmail-category-label-id';
|
||||
import { MESSAGING_GMAIL_EXCLUDED_CATEGORIES } from 'src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-excluded-categories';
|
||||
import { MessagingGmailHistoryService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-history.service';
|
||||
import { computeGmailCategoryLabelId } from 'src/modules/messaging/message-import-manager/drivers/gmail/utils/compute-gmail-category-label-id';
|
||||
import { assertNotNull } from 'src/utils/assert';
|
||||
|
||||
@Injectable()
|
||||
export class GmailFetchMessageIdsToExcludeService {
|
||||
export class MessagingGmailFetchMessageIdsToExcludeService {
|
||||
constructor(
|
||||
private readonly gmailGetHistoryService: GmailGetHistoryService,
|
||||
private readonly gmailGetHistoryService: MessagingGmailHistoryService,
|
||||
) {}
|
||||
|
||||
public async fetchEmailIdsToExcludeOrThrow(
|
||||
@ -19,7 +19,7 @@ export class GmailFetchMessageIdsToExcludeService {
|
||||
): Promise<string[]> {
|
||||
const emailIds: string[] = [];
|
||||
|
||||
for (const category of GMAIL_EXCLUDED_CATEGORIES) {
|
||||
for (const category of MESSAGING_GMAIL_EXCLUDED_CATEGORIES) {
|
||||
const { history, error } = await this.gmailGetHistoryService.getHistory(
|
||||
gmailClient,
|
||||
lastSyncHistoryId,
|
||||
@ -9,27 +9,29 @@ import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decora
|
||||
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { GMAIL_USERS_MESSAGES_LIST_MAX_RESULT } from 'src/modules/messaging/constants/gmail-users-messages-list-max-result.constant';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { GmailClientProvider } from 'src/modules/messaging/services/providers/gmail/gmail-client.provider';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import {
|
||||
GmailError,
|
||||
GmailErrorHandlingService,
|
||||
} from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service';
|
||||
import { MessageChannelSyncStatusService } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.service';
|
||||
import { computeGmailCategoryExcludeSearchFilter } from 'src/modules/messaging/utils/compute-gmail-category-excude-search-filter';
|
||||
import { GMAIL_EXCLUDED_CATEGORIES } from 'src/modules/messaging/constants/gmail-excluded-categories';
|
||||
MessagingErrorHandlingService,
|
||||
} from 'src/modules/messaging/common/services/messaging-error-handling.service';
|
||||
import { computeGmailCategoryExcludeSearchFilter } from 'src/modules/messaging/message-import-manager/drivers/gmail/utils/compute-gmail-category-excude-search-filter';
|
||||
import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
|
||||
import { MessagingGmailClientProvider } from 'src/modules/messaging/message-import-manager/drivers/gmail/providers/messaging-gmail-client.provider';
|
||||
import { MESSAGING_GMAIL_USERS_MESSAGES_LIST_MAX_RESULT } from 'src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-users-messages-list-max-result.constant';
|
||||
import { MESSAGING_GMAIL_EXCLUDED_CATEGORIES } from 'src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-excluded-categories';
|
||||
|
||||
@Injectable()
|
||||
export class GmailFullMessageListFetchV2Service {
|
||||
private readonly logger = new Logger(GmailFullMessageListFetchV2Service.name);
|
||||
export class MessagingGmailFullMessageListFetchService {
|
||||
private readonly logger = new Logger(
|
||||
MessagingGmailFullMessageListFetchService.name,
|
||||
);
|
||||
|
||||
constructor(
|
||||
private readonly gmailClientProvider: GmailClientProvider,
|
||||
private readonly gmailClientProvider: MessagingGmailClientProvider,
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
@InjectCacheStorage(CacheStorageNamespace.Messaging)
|
||||
@ -38,8 +40,8 @@ export class GmailFullMessageListFetchV2Service {
|
||||
MessageChannelMessageAssociationWorkspaceEntity,
|
||||
)
|
||||
private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository,
|
||||
private readonly messageChannelSyncStatusService: MessageChannelSyncStatusService,
|
||||
private readonly gmailErrorHandlingService: GmailErrorHandlingService,
|
||||
private readonly messagingChannelSyncStatusService: MessagingChannelSyncStatusService,
|
||||
private readonly gmailErrorHandlingService: MessagingErrorHandlingService,
|
||||
) {}
|
||||
|
||||
public async processMessageListFetch(
|
||||
@ -47,7 +49,7 @@ export class GmailFullMessageListFetchV2Service {
|
||||
connectedAccount: ObjectRecord<ConnectedAccountWorkspaceEntity>,
|
||||
workspaceId: string,
|
||||
) {
|
||||
await this.messageChannelSyncStatusService.markAsMessagesListFetchOngoing(
|
||||
await this.messagingChannelSyncStatusService.markAsMessagesListFetchOngoing(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -75,7 +77,7 @@ export class GmailFullMessageListFetchV2Service {
|
||||
return;
|
||||
}
|
||||
|
||||
await this.messageChannelSyncStatusService.scheduleMessagesImport(
|
||||
await this.messagingChannelSyncStatusService.scheduleMessagesImport(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -97,9 +99,11 @@ export class GmailFullMessageListFetchV2Service {
|
||||
try {
|
||||
response = await gmailClient.users.messages.list({
|
||||
userId: 'me',
|
||||
maxResults: GMAIL_USERS_MESSAGES_LIST_MAX_RESULT,
|
||||
maxResults: MESSAGING_GMAIL_USERS_MESSAGES_LIST_MAX_RESULT,
|
||||
pageToken,
|
||||
q: computeGmailCategoryExcludeSearchFilter(GMAIL_EXCLUDED_CATEGORIES),
|
||||
q: computeGmailCategoryExcludeSearchFilter(
|
||||
MESSAGING_GMAIL_EXCLUDED_CATEGORIES,
|
||||
),
|
||||
});
|
||||
} catch (error) {
|
||||
return {
|
||||
@ -3,11 +3,11 @@ import { Injectable } from '@nestjs/common';
|
||||
import { gmail_v1 } from 'googleapis';
|
||||
import { GaxiosResponse } from 'gaxios';
|
||||
|
||||
import { GMAIL_USERS_HISTORY_MAX_RESULT } from 'src/modules/messaging/constants/gmail-users-history-max-result.constant';
|
||||
import { GmailError } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service';
|
||||
import { GmailError } from 'src/modules/messaging/common/services/messaging-error-handling.service';
|
||||
import { MESSAGING_GMAIL_USERS_HISTORY_MAX_RESULT } from 'src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-users-history-max-result.constant';
|
||||
|
||||
@Injectable()
|
||||
export class GmailGetHistoryService {
|
||||
export class MessagingGmailHistoryService {
|
||||
constructor() {}
|
||||
|
||||
public async getHistory(
|
||||
@ -30,7 +30,7 @@ export class GmailGetHistoryService {
|
||||
try {
|
||||
response = await gmailClient.users.history.list({
|
||||
userId: 'me',
|
||||
maxResults: GMAIL_USERS_HISTORY_MAX_RESULT,
|
||||
maxResults: MESSAGING_GMAIL_USERS_HISTORY_MAX_RESULT,
|
||||
pageToken,
|
||||
startHistoryId: lastSyncHistoryId,
|
||||
historyTypes: historyTypes || ['messageAdded', 'messageDeleted'],
|
||||
@ -1,39 +1,40 @@
|
||||
import { Injectable, Logger } from '@nestjs/common';
|
||||
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { FetchMessagesByBatchesService } from 'src/modules/messaging/services/fetch-messages-by-batches/fetch-messages-by-batches.service';
|
||||
import {
|
||||
MessageChannelWorkspaceEntity,
|
||||
MessageChannelSyncSubStatus,
|
||||
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { createQueriesFromMessageIds } from 'src/modules/messaging/utils/create-queries-from-message-ids.util';
|
||||
import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator';
|
||||
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
|
||||
import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache-storage.service';
|
||||
import { GMAIL_USERS_MESSAGES_GET_BATCH_SIZE } from 'src/modules/messaging/constants/gmail-users-messages-get-batch-size.constant';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { SaveMessagesAndEnqueueContactCreationService } from 'src/modules/messaging/services/gmail-messages-import/save-messages-and-enqueue-contact-creation.service';
|
||||
import { GmailErrorHandlingService } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service';
|
||||
import { MessageChannelSyncStatusService } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.service';
|
||||
import { GoogleAPIRefreshAccessTokenService } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.service';
|
||||
import { GmailMessagesImportService } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import.service';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/services/telemetry/messaging-telemetry.service';
|
||||
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { BlocklistRepository } from 'src/modules/connected-account/repositories/blocklist.repository';
|
||||
import { filterEmails } from 'src/modules/messaging/services/utils/filter-emails.util';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
|
||||
import {
|
||||
MessageChannelWorkspaceEntity,
|
||||
MessageChannelSyncSubStatus,
|
||||
} from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { createQueriesFromMessageIds } from 'src/modules/messaging/message-import-manager/utils/create-queries-from-message-ids.util';
|
||||
import { filterEmails } from 'src/modules/messaging/message-import-manager/utils/filter-emails.util';
|
||||
import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
|
||||
import { MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE } from 'src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-users-messages-get-batch-size.constant';
|
||||
import { MessagingGmailFetchMessagesByBatchesService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-fetch-messages-by-batches.service';
|
||||
import { MessagingErrorHandlingService } from 'src/modules/messaging/common/services/messaging-error-handling.service';
|
||||
import { MessagingSaveMessagesAndEnqueueContactCreationService } from 'src/modules/messaging/common/services/messaging-save-messages-and-enqueue-contact-creation.service';
|
||||
|
||||
@Injectable()
|
||||
export class GmailMessagesImportV2Service {
|
||||
private readonly logger = new Logger(GmailMessagesImportService.name);
|
||||
export class MessagingGmailMessagesImportService {
|
||||
private readonly logger = new Logger(
|
||||
MessagingGmailMessagesImportService.name,
|
||||
);
|
||||
|
||||
constructor(
|
||||
private readonly fetchMessagesByBatchesService: FetchMessagesByBatchesService,
|
||||
private readonly fetchMessagesByBatchesService: MessagingGmailFetchMessagesByBatchesService,
|
||||
@InjectCacheStorage(CacheStorageNamespace.Messaging)
|
||||
private readonly cacheStorage: CacheStorageService,
|
||||
private readonly messageChannelSyncStatusService: MessageChannelSyncStatusService,
|
||||
private readonly saveMessagesAndEnqueueContactCreationService: SaveMessagesAndEnqueueContactCreationService,
|
||||
private readonly gmailErrorHandlingService: GmailErrorHandlingService,
|
||||
private readonly messagingChannelSyncStatusService: MessagingChannelSyncStatusService,
|
||||
private readonly saveMessagesAndEnqueueContactCreationService: MessagingSaveMessagesAndEnqueueContactCreationService,
|
||||
private readonly gmailErrorHandlingService: MessagingErrorHandlingService,
|
||||
private readonly googleAPIsRefreshAccessTokenService: GoogleAPIRefreshAccessTokenService,
|
||||
private readonly messagingTelemetryService: MessagingTelemetryService,
|
||||
@InjectObjectMetadataRepository(BlocklistWorkspaceEntity)
|
||||
@ -63,7 +64,7 @@ export class GmailMessagesImportV2Service {
|
||||
`Messaging import for workspace ${workspaceId} and account ${connectedAccount.id} starting...`,
|
||||
);
|
||||
|
||||
await this.messageChannelSyncStatusService.markAsMessagesImportOngoing(
|
||||
await this.messagingChannelSyncStatusService.markAsMessagesImportOngoing(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -76,11 +77,11 @@ export class GmailMessagesImportV2Service {
|
||||
const messageIdsToFetch =
|
||||
(await this.cacheStorage.setPop(
|
||||
`messages-to-import:${workspaceId}:gmail:${messageChannel.id}`,
|
||||
GMAIL_USERS_MESSAGES_GET_BATCH_SIZE,
|
||||
MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE,
|
||||
)) ?? [];
|
||||
|
||||
if (!messageIdsToFetch?.length) {
|
||||
await this.messageChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch(
|
||||
await this.messagingChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -119,13 +120,15 @@ export class GmailMessagesImportV2Service {
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
if (messageIdsToFetch.length < GMAIL_USERS_MESSAGES_GET_BATCH_SIZE) {
|
||||
await this.messageChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch(
|
||||
if (
|
||||
messageIdsToFetch.length < MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE
|
||||
) {
|
||||
await this.messagingChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
} else {
|
||||
await this.messageChannelSyncStatusService.scheduleMessagesImport(
|
||||
await this.messagingChannelSyncStatusService.scheduleMessagesImport(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -2,30 +2,30 @@ import { Injectable, Logger } from '@nestjs/common';
|
||||
|
||||
import { gmail_v1 } from 'googleapis';
|
||||
|
||||
import { GmailClientProvider } from 'src/modules/messaging/services/providers/gmail/gmail-client.provider';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache-storage.service';
|
||||
import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator';
|
||||
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
|
||||
import { GmailGetHistoryService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-get-history.service';
|
||||
import { ObjectRecord } from 'src/engine/workspace-manager/workspace-sync-metadata/types/object-record';
|
||||
import { GmailErrorHandlingService } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service';
|
||||
import { MessageChannelSyncStatusService } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.service';
|
||||
import { GmailFetchMessageIdsToExcludeService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-fetch-messages-ids-to-exclude.service';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/common/repositories/message-channel-message-association.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { MessagingGmailClientProvider } from 'src/modules/messaging/message-import-manager/drivers/gmail/providers/messaging-gmail-client.provider';
|
||||
import { MessagingChannelSyncStatusService } from 'src/modules/messaging/common/services/messaging-channel-sync-status.service';
|
||||
import { MessagingGmailFetchMessageIdsToExcludeService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-fetch-messages-ids-to-exclude.service';
|
||||
import { MessagingErrorHandlingService } from 'src/modules/messaging/common/services/messaging-error-handling.service';
|
||||
import { MessagingGmailHistoryService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-history.service';
|
||||
|
||||
@Injectable()
|
||||
export class GmailPartialMessageListFetchV2Service {
|
||||
export class MessagingGmailPartialMessageListFetchService {
|
||||
private readonly logger = new Logger(
|
||||
GmailPartialMessageListFetchV2Service.name,
|
||||
MessagingGmailPartialMessageListFetchService.name,
|
||||
);
|
||||
|
||||
constructor(
|
||||
private readonly gmailClientProvider: GmailClientProvider,
|
||||
private readonly gmailClientProvider: MessagingGmailClientProvider,
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
@InjectCacheStorage(CacheStorageNamespace.Messaging)
|
||||
@ -34,10 +34,10 @@ export class GmailPartialMessageListFetchV2Service {
|
||||
MessageChannelMessageAssociationWorkspaceEntity,
|
||||
)
|
||||
private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository,
|
||||
private readonly gmailErrorHandlingService: GmailErrorHandlingService,
|
||||
private readonly gmailGetHistoryService: GmailGetHistoryService,
|
||||
private readonly messageChannelSyncStatusService: MessageChannelSyncStatusService,
|
||||
private readonly gmailFetchMessageIdsToExcludeService: GmailFetchMessageIdsToExcludeService,
|
||||
private readonly gmailErrorHandlingService: MessagingErrorHandlingService,
|
||||
private readonly gmailGetHistoryService: MessagingGmailHistoryService,
|
||||
private readonly messagingChannelSyncStatusService: MessagingChannelSyncStatusService,
|
||||
private readonly gmailFetchMessageIdsToExcludeService: MessagingGmailFetchMessageIdsToExcludeService,
|
||||
) {}
|
||||
|
||||
public async processMessageListFetch(
|
||||
@ -45,7 +45,7 @@ export class GmailPartialMessageListFetchV2Service {
|
||||
connectedAccount: ObjectRecord<ConnectedAccountWorkspaceEntity>,
|
||||
workspaceId: string,
|
||||
): Promise<void> {
|
||||
await this.messageChannelSyncStatusService.markAsMessagesListFetchOngoing(
|
||||
await this.messagingChannelSyncStatusService.markAsMessagesListFetchOngoing(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -85,7 +85,7 @@ export class GmailPartialMessageListFetchV2Service {
|
||||
`Partial message list import done with history ${historyId} and nothing to update for workspace ${workspaceId} and account ${connectedAccount.id}`,
|
||||
);
|
||||
|
||||
await this.messageChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch(
|
||||
await this.messagingChannelSyncStatusService.markAsCompletedAndSchedulePartialMessageListFetch(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -144,7 +144,7 @@ export class GmailPartialMessageListFetchV2Service {
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
await this.messageChannelSyncStatusService.scheduleMessagesImport(
|
||||
await this.messagingChannelSyncStatusService.scheduleMessagesImport(
|
||||
messageChannel.id,
|
||||
workspaceId,
|
||||
);
|
||||
@ -1,4 +1,4 @@
|
||||
import { computeGmailCategoryExcludeSearchFilter } from 'src/modules/messaging/utils/compute-gmail-category-excude-search-filter';
|
||||
import { computeGmailCategoryExcludeSearchFilter } from 'src/modules/messaging/message-import-manager/drivers/gmail/utils/compute-gmail-category-excude-search-filter';
|
||||
|
||||
describe('computeGmailCategoryExcludeSearchFilter', () => {
|
||||
it('should return correct exclude search filter with empty category array', () => {
|
||||
@ -1,4 +1,4 @@
|
||||
import { computeGmailCategoryLabelId } from 'src/modules/messaging/utils/compute-gmail-category-label-id';
|
||||
import { computeGmailCategoryLabelId } from 'src/modules/messaging/message-import-manager/drivers/gmail/utils/compute-gmail-category-label-id';
|
||||
|
||||
describe('computeGmailCategoryLabelId', () => {
|
||||
it('should return correct category label id', () => {
|
||||
@ -2,32 +2,32 @@ import { Injectable, Logger } from '@nestjs/common';
|
||||
|
||||
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
|
||||
|
||||
import { GmailPartialMessageListFetchV2Service } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-partial-message-list-fetch-v2.service';
|
||||
import {
|
||||
MessageChannelSyncSubStatus,
|
||||
MessageChannelWorkspaceEntity,
|
||||
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { GmailFullMessageListFetchV2Service } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch-v2.service';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/services/telemetry/messaging-telemetry.service';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
|
||||
import {
|
||||
MessageChannelSyncSubStatus,
|
||||
MessageChannelWorkspaceEntity,
|
||||
} from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { MessagingGmailFullMessageListFetchService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-full-message-list-fetch.service';
|
||||
import { MessagingGmailPartialMessageListFetchService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-partial-message-list-fetch.service';
|
||||
|
||||
export type GmailMessageListFetchJobData = {
|
||||
export type MessagingMessageListFetchJobData = {
|
||||
workspaceId: string;
|
||||
connectedAccountId: string;
|
||||
};
|
||||
|
||||
@Injectable()
|
||||
export class GmailMessageListFetchJob
|
||||
implements MessageQueueJob<GmailMessageListFetchJobData>
|
||||
export class MessagingMessageListFetchJob
|
||||
implements MessageQueueJob<MessagingMessageListFetchJobData>
|
||||
{
|
||||
private readonly logger = new Logger(GmailMessageListFetchJob.name);
|
||||
private readonly logger = new Logger(MessagingMessageListFetchJob.name);
|
||||
|
||||
constructor(
|
||||
private readonly gmailFullMessageListFetchV2Service: GmailFullMessageListFetchV2Service,
|
||||
private readonly gmailPartialMessageListFetchV2Service: GmailPartialMessageListFetchV2Service,
|
||||
private readonly gmailFullMessageListFetchService: MessagingGmailFullMessageListFetchService,
|
||||
private readonly gmailPartialMessageListFetchV2Service: MessagingGmailPartialMessageListFetchService,
|
||||
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
|
||||
private readonly connectedAccountRepository: ConnectedAccountRepository,
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
@ -35,7 +35,7 @@ export class GmailMessageListFetchJob
|
||||
private readonly messagingTelemetryService: MessagingTelemetryService,
|
||||
) {}
|
||||
|
||||
async handle(data: GmailMessageListFetchJobData): Promise<void> {
|
||||
async handle(data: MessagingMessageListFetchJobData): Promise<void> {
|
||||
const { workspaceId, connectedAccountId } = data;
|
||||
|
||||
await this.messagingTelemetryService.track({
|
||||
@ -113,7 +113,7 @@ export class GmailMessageListFetchJob
|
||||
connectedAccountId,
|
||||
});
|
||||
|
||||
await this.gmailFullMessageListFetchV2Service.processMessageListFetch(
|
||||
await this.gmailFullMessageListFetchService.processMessageListFetch(
|
||||
messageChannel,
|
||||
connectedAccount,
|
||||
workspaceId,
|
||||
@ -0,0 +1,61 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
import { MessageQueueJob } from 'src/engine/integrations/message-queue/interfaces/message-queue-job.interface';
|
||||
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import { MessagingTelemetryService } from 'src/modules/messaging/common/services/messaging-telemetry.service';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { MessagingGmailMessagesImportService } from 'src/modules/messaging/message-import-manager/drivers/gmail/services/messaging-gmail-messages-import.service';
|
||||
|
||||
export type MessagingMessagesImportJobData = {
|
||||
workspaceId: string;
|
||||
};
|
||||
|
||||
@Injectable()
|
||||
export class MessagingMessagesImportJob
|
||||
implements MessageQueueJob<MessagingMessagesImportJobData>
|
||||
{
|
||||
constructor(
|
||||
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
|
||||
private readonly connectedAccountRepository: ConnectedAccountRepository,
|
||||
private readonly gmailFetchMessageContentFromCacheService: MessagingGmailMessagesImportService,
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
private readonly messagingTelemetryService: MessagingTelemetryService,
|
||||
) {}
|
||||
|
||||
async handle(data: MessagingMessagesImportJobData): Promise<void> {
|
||||
const { workspaceId } = data;
|
||||
|
||||
const messageChannels =
|
||||
await this.messageChannelRepository.getAll(workspaceId);
|
||||
|
||||
for (const messageChannel of messageChannels) {
|
||||
if (!messageChannel?.isSyncEnabled) {
|
||||
continue;
|
||||
}
|
||||
|
||||
await this.messagingTelemetryService.track({
|
||||
eventName: 'messages_import.triggered',
|
||||
workspaceId,
|
||||
connectedAccountId: messageChannel.connectedAccountId,
|
||||
messageChannelId: messageChannel.id,
|
||||
});
|
||||
|
||||
const connectedAccount =
|
||||
await this.connectedAccountRepository.getConnectedAccountOrThrow(
|
||||
workspaceId,
|
||||
messageChannel.connectedAccountId,
|
||||
);
|
||||
|
||||
await this.gmailFetchMessageContentFromCacheService.processMessageBatchImport(
|
||||
messageChannel,
|
||||
connectedAccount,
|
||||
workspaceId,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,44 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { Workspace } from 'src/engine/core-modules/workspace/workspace.entity';
|
||||
import { DataSourceEntity } from 'src/engine/metadata-modules/data-source/data-source.entity';
|
||||
import { MessagingCommonModule } from 'src/modules/messaging/common/messaging-common.module';
|
||||
import { MessagingMessageListFetchCronCommand } from 'src/modules/messaging/message-import-manager/crons/commands/messaging-message-list-fetch.cron.command';
|
||||
import { MessagingMessagesImportCronCommand } from 'src/modules/messaging/message-import-manager/crons/commands/messaging-messages-import.cron.command';
|
||||
import { MessagingMessageListFetchCronJob } from 'src/modules/messaging/message-import-manager/crons/jobs/messaging-message-list-fetch.cron.job';
|
||||
import { MessagingMessagesImportCronJob } from 'src/modules/messaging/message-import-manager/crons/jobs/messaging-messages-import.cron.job';
|
||||
import { MessagingGmailDriverModule } from 'src/modules/messaging/message-import-manager/drivers/gmail/messaging-gmail-driver.module';
|
||||
import { MessagingMessageListFetchJob } from 'src/modules/messaging/message-import-manager/jobs/messaging-message-list-fetch.job';
|
||||
import { MessagingMessagesImportJob } from 'src/modules/messaging/message-import-manager/jobs/messaging-messages-import.job';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
MessagingGmailDriverModule,
|
||||
MessagingCommonModule,
|
||||
TypeOrmModule.forFeature([Workspace], 'core'),
|
||||
TypeOrmModule.forFeature([DataSourceEntity], 'metadata'),
|
||||
],
|
||||
providers: [
|
||||
MessagingMessageListFetchCronCommand,
|
||||
MessagingMessagesImportCronCommand,
|
||||
{
|
||||
provide: MessagingMessageListFetchJob.name,
|
||||
useClass: MessagingMessageListFetchJob,
|
||||
},
|
||||
{
|
||||
provide: MessagingMessagesImportJob.name,
|
||||
useClass: MessagingMessagesImportJob,
|
||||
},
|
||||
{
|
||||
provide: MessagingMessageListFetchCronJob.name,
|
||||
useClass: MessagingMessageListFetchCronJob,
|
||||
},
|
||||
{
|
||||
provide: MessagingMessagesImportCronJob.name,
|
||||
useClass: MessagingMessagesImportCronJob,
|
||||
},
|
||||
],
|
||||
exports: [],
|
||||
})
|
||||
export class MessagingImportManagerModule {}
|
||||
@ -1,4 +1,4 @@
|
||||
import { formatAddressObjectAsParticipants } from 'src/modules/messaging/services/utils/format-address-object-as-participants.util';
|
||||
import { formatAddressObjectAsParticipants } from 'src/modules/messaging/message-import-manager/utils/format-address-object-as-participants.util';
|
||||
|
||||
describe('formatAddressObjectAsParticipants', () => {
|
||||
it('should format address object as participants', () => {
|
||||
@ -1,4 +1,4 @@
|
||||
import { MessageQuery } from 'src/modules/messaging/types/message-or-thread-query';
|
||||
import { MessageQuery } from 'src/modules/messaging/message-import-manager/types/message-or-thread-query';
|
||||
|
||||
export const createQueriesFromMessageIds = (
|
||||
messageExternalIds: string[],
|
||||
@ -1,5 +1,5 @@
|
||||
import { GmailMessage } from 'src/modules/messaging/types/gmail-message';
|
||||
import { isEmailBlocklisted } from 'src/modules/calendar-messaging-participant/utils/is-email-blocklisted.util';
|
||||
import { GmailMessage } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
|
||||
|
||||
// Todo: refactor this into several utils
|
||||
export const filterEmails = (messages: GmailMessage[], blocklist: string[]) => {
|
||||
@ -1,6 +1,6 @@
|
||||
import addressparser from 'addressparser';
|
||||
|
||||
import { Participant } from 'src/modules/messaging/types/gmail-message';
|
||||
import { Participant } from 'src/modules/messaging/message-import-manager/drivers/gmail/types/gmail-message';
|
||||
|
||||
const formatAddressObjectAsArray = (
|
||||
addressObject: addressparser.EmailAddress | addressparser.EmailAddress[],
|
||||
@ -11,10 +11,10 @@ import {
|
||||
} from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { CreateCompanyAndContactService } from 'src/modules/connected-account/auto-companies-and-contacts-creation/services/create-company-and-contact.service';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { MessageParticipantRepository } from 'src/modules/messaging/repositories/message-participant.repository';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-participant.workspace-entity';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/common/repositories/message-channel.repository';
|
||||
import { MessageParticipantRepository } from 'src/modules/messaging/common/repositories/message-participant.repository';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-channel.workspace-entity';
|
||||
import { MessageParticipantWorkspaceEntity } from 'src/modules/messaging/common/standard-objects/message-participant.workspace-entity';
|
||||
|
||||
export type MessagingCreateCompanyAndContactAfterSyncJobData = {
|
||||
workspaceId: string;
|
||||
@ -0,0 +1,24 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { AnalyticsModule } from 'src/engine/core-modules/analytics/analytics.module';
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { AutoCompaniesAndContactsCreationModule } from 'src/modules/connected-account/auto-companies-and-contacts-creation/auto-companies-and-contacts-creation.module';
|
||||
import { MessagingGmailDriverModule } from 'src/modules/messaging/message-import-manager/drivers/gmail/messaging-gmail-driver.module';
|
||||
import { MessagingCreateCompanyAndContactAfterSyncJob } from 'src/modules/messaging/message-participants-manager/jobs/messaging-create-company-and-contact-after-sync.job';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
|
||||
AnalyticsModule,
|
||||
MessagingGmailDriverModule,
|
||||
AutoCompaniesAndContactsCreationModule,
|
||||
],
|
||||
providers: [
|
||||
{
|
||||
provide: MessagingCreateCompanyAndContactAfterSyncJob.name,
|
||||
useClass: MessagingCreateCompanyAndContactAfterSyncJob,
|
||||
},
|
||||
],
|
||||
})
|
||||
export class MessaginParticipantsManagerModule {}
|
||||
@ -1,22 +1,18 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { MessagingMessageChannelListener } from 'src/modules/messaging/listeners/messaging-message-channel.listener';
|
||||
import { MessagingConnectedAccountListener } from 'src/modules/messaging/listeners/messaging-connected-account.listener';
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { ParticipantPersonListener } from 'src/modules/calendar-messaging-participant/listeners/participant-person.listener';
|
||||
import { ParticipantWorkspaceMemberListener } from 'src/modules/calendar-messaging-participant/listeners/participant-workspace-member.listener';
|
||||
import { MessagingBlocklistListener } from 'src/modules/messaging/listeners/messaging-blocklist.listener';
|
||||
import { MessagingBlocklistManagerModule } from 'src/modules/messaging/blocklist-manager/messaging-blocklist-manager.module';
|
||||
import { MessagingMessageCleanerModule } from 'src/modules/messaging/message-cleaner/messaging-message-cleaner.module';
|
||||
import { MessagingImportManagerModule } from 'src/modules/messaging/message-import-manager/messaging-import-manager.module';
|
||||
import { MessaginParticipantsManagerModule } from 'src/modules/messaging/message-participants-manager/messaging-participants-manager.module';
|
||||
|
||||
@Module({
|
||||
imports: [TypeOrmModule.forFeature([FeatureFlagEntity], 'core')],
|
||||
providers: [
|
||||
ParticipantPersonListener,
|
||||
ParticipantWorkspaceMemberListener,
|
||||
MessagingMessageChannelListener,
|
||||
MessagingConnectedAccountListener,
|
||||
MessagingBlocklistListener,
|
||||
imports: [
|
||||
MessagingImportManagerModule,
|
||||
MessagingMessageCleanerModule,
|
||||
MessaginParticipantsManagerModule,
|
||||
MessagingBlocklistManagerModule,
|
||||
],
|
||||
providers: [],
|
||||
exports: [],
|
||||
})
|
||||
export class MessagingModule {}
|
||||
|
||||
@ -1,15 +0,0 @@
|
||||
import { HttpModule } from '@nestjs/axios';
|
||||
import { Module } from '@nestjs/common';
|
||||
|
||||
import { FetchByBatchesService } from 'src/modules/messaging/services/fetch-by-batch/fetch-by-batch.service';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
HttpModule.register({
|
||||
baseURL: 'https://www.googleapis.com/batch/gmail/v1',
|
||||
}),
|
||||
],
|
||||
providers: [FetchByBatchesService],
|
||||
exports: [FetchByBatchesService],
|
||||
})
|
||||
export class FetchByBatchModule {}
|
||||
@ -1,11 +0,0 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
|
||||
import { FetchByBatchModule } from 'src/modules/messaging/services/fetch-by-batch/fetch-by-batch.module';
|
||||
import { FetchMessagesByBatchesService } from 'src/modules/messaging/services/fetch-messages-by-batches/fetch-messages-by-batches.service';
|
||||
|
||||
@Module({
|
||||
imports: [FetchByBatchModule],
|
||||
providers: [FetchMessagesByBatchesService],
|
||||
exports: [FetchMessagesByBatchesService],
|
||||
})
|
||||
export class FetchMessagesByBatchesModule {}
|
||||
@ -1,12 +0,0 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
|
||||
import { GmailErrorHandlingService } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.service';
|
||||
import { SetMessageChannelSyncStatusModule } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.module';
|
||||
import { MessagingTelemetryModule } from 'src/modules/messaging/services/telemetry/messaging-telemetry.module';
|
||||
|
||||
@Module({
|
||||
imports: [SetMessageChannelSyncStatusModule, MessagingTelemetryModule],
|
||||
providers: [GmailErrorHandlingService],
|
||||
exports: [GmailErrorHandlingService],
|
||||
})
|
||||
export class GmailErrorHandlingModule {}
|
||||
@ -1,42 +0,0 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { WorkspaceDataSourceModule } from 'src/engine/workspace-datasource/workspace-datasource.module';
|
||||
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { FetchMessagesByBatchesModule } from 'src/modules/messaging/services/fetch-messages-by-batches/fetch-messages-by-batches.module';
|
||||
import { GmailErrorHandlingModule } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.module';
|
||||
import { GmailFullMessageListFetchV2Service } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch-v2.service';
|
||||
import { GmailFullMessageListFetchService } from 'src/modules/messaging/services/gmail-full-message-list-fetch/gmail-full-message-list-fetch.service';
|
||||
import { SetMessageChannelSyncStatusModule } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.module';
|
||||
import { MessagingProvidersModule } from 'src/modules/messaging/services/providers/messaging-providers.module';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
MessagingProvidersModule,
|
||||
FetchMessagesByBatchesModule,
|
||||
GmailErrorHandlingModule,
|
||||
ObjectMetadataRepositoryModule.forFeature([
|
||||
ConnectedAccountWorkspaceEntity,
|
||||
MessageChannelWorkspaceEntity,
|
||||
MessageChannelMessageAssociationWorkspaceEntity,
|
||||
BlocklistWorkspaceEntity,
|
||||
]),
|
||||
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
|
||||
WorkspaceDataSourceModule,
|
||||
SetMessageChannelSyncStatusModule,
|
||||
],
|
||||
providers: [
|
||||
GmailFullMessageListFetchService,
|
||||
GmailFullMessageListFetchV2Service,
|
||||
],
|
||||
exports: [
|
||||
GmailFullMessageListFetchService,
|
||||
GmailFullMessageListFetchV2Service,
|
||||
],
|
||||
})
|
||||
export class GmailFullMessageListFetchModule {}
|
||||
@ -1,306 +0,0 @@
|
||||
import { Injectable, Logger } from '@nestjs/common';
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { EntityManager, Repository } from 'typeorm';
|
||||
import { gmail_v1 } from 'googleapis';
|
||||
|
||||
import {
|
||||
FeatureFlagEntity,
|
||||
FeatureFlagKeys,
|
||||
} from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache-storage.service';
|
||||
import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator';
|
||||
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { BlocklistRepository } from 'src/modules/connected-account/repositories/blocklist.repository';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { GMAIL_USERS_MESSAGES_LIST_MAX_RESULT } from 'src/modules/messaging/constants/gmail-users-messages-list-max-result.constant';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { GmailClientProvider } from 'src/modules/messaging/services/providers/gmail/gmail-client.provider';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import {
|
||||
MessageChannelWorkspaceEntity,
|
||||
MessageChannelSyncStatus,
|
||||
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import { computeGmailCategoryExcludeSearchFilter } from 'src/modules/messaging/utils/compute-gmail-category-excude-search-filter';
|
||||
import { GMAIL_EXCLUDED_CATEGORIES } from 'src/modules/messaging/constants/gmail-excluded-categories';
|
||||
|
||||
@Injectable()
|
||||
export class GmailFullMessageListFetchService {
|
||||
private readonly logger = new Logger(GmailFullMessageListFetchService.name);
|
||||
|
||||
constructor(
|
||||
private readonly gmailClientProvider: GmailClientProvider,
|
||||
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
|
||||
private readonly connectedAccountRepository: ConnectedAccountRepository,
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
@InjectObjectMetadataRepository(BlocklistWorkspaceEntity)
|
||||
private readonly blocklistRepository: BlocklistRepository,
|
||||
@InjectRepository(FeatureFlagEntity, 'core')
|
||||
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
|
||||
@InjectCacheStorage(CacheStorageNamespace.Messaging)
|
||||
private readonly cacheStorage: CacheStorageService,
|
||||
@InjectObjectMetadataRepository(
|
||||
MessageChannelMessageAssociationWorkspaceEntity,
|
||||
)
|
||||
private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository,
|
||||
private readonly workspaceDataSourceService: WorkspaceDataSourceService,
|
||||
) {}
|
||||
|
||||
public async fetchConnectedAccountThreads(
|
||||
workspaceId: string,
|
||||
connectedAccountId: string,
|
||||
includedEmails?: string[],
|
||||
) {
|
||||
const connectedAccount = await this.connectedAccountRepository.getById(
|
||||
connectedAccountId,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
if (!connectedAccount) {
|
||||
this.logger.error(
|
||||
`Connected account ${connectedAccountId} not found in workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
const refreshToken = connectedAccount.refreshToken;
|
||||
|
||||
if (!refreshToken) {
|
||||
throw new Error(
|
||||
`No refresh token found for connected account ${connectedAccountId} in workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
|
||||
const gmailMessageChannel =
|
||||
await this.messageChannelRepository.getFirstByConnectedAccountId(
|
||||
connectedAccountId,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
if (!gmailMessageChannel) {
|
||||
this.logger.error(
|
||||
`No message channel found for connected account ${connectedAccountId} in workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
if (gmailMessageChannel.syncStatus === MessageChannelSyncStatus.ONGOING) {
|
||||
this.logger.log(
|
||||
`Messaging import for workspace ${workspaceId} and account ${connectedAccountId} is locked, import will be retried later.`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.ONGOING,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
const workspaceDataSource =
|
||||
await this.workspaceDataSourceService.connectToWorkspaceDataSource(
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
const gmailClient: gmail_v1.Gmail =
|
||||
await this.gmailClientProvider.getGmailClient(refreshToken);
|
||||
|
||||
const blocklistedEmails = await this.fetchBlocklistEmails(
|
||||
connectedAccount.accountOwnerId,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
await workspaceDataSource
|
||||
?.transaction(async (transactionManager) => {
|
||||
await this.fetchAllMessageIdsFromGmailAndStoreInCache(
|
||||
gmailClient,
|
||||
gmailMessageChannel.id,
|
||||
includedEmails || [],
|
||||
blocklistedEmails,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.PENDING,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
})
|
||||
.catch(async (error) => {
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.FAILED,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
throw new Error(
|
||||
`Error fetching messages for ${connectedAccountId} in workspace ${workspaceId}: ${error.message}`,
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
public async fetchAllMessageIdsFromGmailAndStoreInCache(
|
||||
gmailClient: gmail_v1.Gmail,
|
||||
messageChannelId: string,
|
||||
includedEmails: string[],
|
||||
blocklistedEmails: string[],
|
||||
workspaceId: string,
|
||||
transactionManager?: EntityManager,
|
||||
) {
|
||||
let pageToken: string | undefined;
|
||||
let hasMoreMessages = true;
|
||||
let messageIdsToFetch = 0;
|
||||
let firstMessageExternalId;
|
||||
|
||||
while (hasMoreMessages) {
|
||||
const response = await gmailClient.users.messages.list({
|
||||
userId: 'me',
|
||||
maxResults: GMAIL_USERS_MESSAGES_LIST_MAX_RESULT,
|
||||
pageToken,
|
||||
q: computeGmailCategoryExcludeSearchFilter(GMAIL_EXCLUDED_CATEGORIES),
|
||||
});
|
||||
|
||||
if (response.data?.messages) {
|
||||
const messageExternalIds = response.data.messages
|
||||
.filter((message): message is { id: string } => message.id != null)
|
||||
.map((message) => message.id);
|
||||
|
||||
if (!firstMessageExternalId) {
|
||||
firstMessageExternalId = messageExternalIds[0];
|
||||
}
|
||||
|
||||
const existingMessageChannelMessageAssociations =
|
||||
await this.messageChannelMessageAssociationRepository.getByMessageExternalIdsAndMessageChannelId(
|
||||
messageExternalIds,
|
||||
messageChannelId,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
const existingMessageChannelMessageAssociationsExternalIds =
|
||||
existingMessageChannelMessageAssociations.map(
|
||||
(messageChannelMessageAssociation) =>
|
||||
messageChannelMessageAssociation.messageExternalId,
|
||||
);
|
||||
|
||||
const messageIdsToImport = messageExternalIds.filter(
|
||||
(messageExternalId) =>
|
||||
!existingMessageChannelMessageAssociationsExternalIds.includes(
|
||||
messageExternalId,
|
||||
),
|
||||
);
|
||||
|
||||
if (messageIdsToImport && messageIdsToImport.length) {
|
||||
await this.cacheStorage.setAdd(
|
||||
`messages-to-import:${workspaceId}:gmail:${messageChannelId}`,
|
||||
messageIdsToImport,
|
||||
);
|
||||
|
||||
messageIdsToFetch += messageIdsToImport.length;
|
||||
}
|
||||
}
|
||||
|
||||
pageToken = response.data.nextPageToken ?? undefined;
|
||||
hasMoreMessages = !!pageToken;
|
||||
}
|
||||
|
||||
if (!messageIdsToFetch) {
|
||||
this.logger.log(
|
||||
`No messages found in Gmail for messageChannel ${messageChannelId} in workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
this.logger.log(
|
||||
`Fetched all ${messageIdsToFetch} message ids from Gmail for messageChannel ${messageChannelId} in workspace ${workspaceId} and added to cache for import`,
|
||||
);
|
||||
|
||||
await this.updateLastSyncCursor(
|
||||
gmailClient,
|
||||
messageChannelId,
|
||||
firstMessageExternalId,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
}
|
||||
|
||||
public async fetchBlocklistEmails(
|
||||
workspaceMemberId: string,
|
||||
workspaceId: string,
|
||||
) {
|
||||
const isBlocklistEnabledFeatureFlag =
|
||||
await this.featureFlagRepository.findOneBy({
|
||||
workspaceId,
|
||||
key: FeatureFlagKeys.IsBlocklistEnabled,
|
||||
value: true,
|
||||
});
|
||||
|
||||
const isBlocklistEnabled =
|
||||
isBlocklistEnabledFeatureFlag && isBlocklistEnabledFeatureFlag.value;
|
||||
|
||||
const blocklist = isBlocklistEnabled
|
||||
? await this.blocklistRepository.getByWorkspaceMemberId(
|
||||
workspaceMemberId,
|
||||
workspaceId,
|
||||
)
|
||||
: [];
|
||||
|
||||
return blocklist.map((blocklist) => blocklist.handle);
|
||||
}
|
||||
|
||||
private async updateLastSyncCursor(
|
||||
gmailClient: gmail_v1.Gmail,
|
||||
messageChannelId: string,
|
||||
firstMessageExternalId: string,
|
||||
workspaceId: string,
|
||||
transactionManager?: EntityManager,
|
||||
) {
|
||||
if (!firstMessageExternalId) {
|
||||
throw new Error(
|
||||
`No first message found for workspace ${workspaceId} and account ${messageChannelId}, can't update sync external id`,
|
||||
);
|
||||
}
|
||||
|
||||
const firstMessageContent = await gmailClient.users.messages.get({
|
||||
userId: 'me',
|
||||
id: firstMessageExternalId,
|
||||
});
|
||||
|
||||
if (!firstMessageContent?.data) {
|
||||
throw new Error(
|
||||
`No first message content found for message ${firstMessageExternalId} in workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
|
||||
const historyId = firstMessageContent?.data?.historyId;
|
||||
|
||||
if (!historyId) {
|
||||
throw new Error(
|
||||
`No historyId found for message ${firstMessageExternalId} in workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
|
||||
this.logger.log(
|
||||
`Updating last external id: ${historyId} for workspace ${workspaceId} and account ${messageChannelId} succeeded.`,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.updateLastSyncCursorIfHigher(
|
||||
messageChannelId,
|
||||
historyId,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
}
|
||||
}
|
||||
@ -1,44 +0,0 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { WorkspaceDataSourceModule } from 'src/engine/workspace-datasource/workspace-datasource.module';
|
||||
import { GoogleAPIRefreshAccessTokenModule } from 'src/modules/connected-account/services/google-api-refresh-access-token/google-api-refresh-access-token.module';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { FetchMessagesByBatchesModule } from 'src/modules/messaging/services/fetch-messages-by-batches/fetch-messages-by-batches.module';
|
||||
import { GmailErrorHandlingModule } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.module';
|
||||
import { GmailMessagesImportV2Service } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import-v2.service';
|
||||
import { GmailMessagesImportService } from 'src/modules/messaging/services/gmail-messages-import/gmail-messages-import.service';
|
||||
import { SaveMessagesAndEnqueueContactCreationService } from 'src/modules/messaging/services/gmail-messages-import/save-messages-and-enqueue-contact-creation.service';
|
||||
import { SetMessageChannelSyncStatusModule } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.module';
|
||||
import { MessageParticipantModule } from 'src/modules/messaging/services/message-participant/message-participant.module';
|
||||
import { MessageModule } from 'src/modules/messaging/services/message/message.module';
|
||||
import { MessagingTelemetryModule } from 'src/modules/messaging/services/telemetry/messaging-telemetry.module';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
FetchMessagesByBatchesModule,
|
||||
ObjectMetadataRepositoryModule.forFeature([
|
||||
ConnectedAccountWorkspaceEntity,
|
||||
MessageChannelWorkspaceEntity,
|
||||
]),
|
||||
MessageModule,
|
||||
WorkspaceDataSourceModule,
|
||||
MessageModule,
|
||||
MessageParticipantModule,
|
||||
SetMessageChannelSyncStatusModule,
|
||||
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
|
||||
GmailErrorHandlingModule,
|
||||
GoogleAPIRefreshAccessTokenModule,
|
||||
MessagingTelemetryModule,
|
||||
],
|
||||
providers: [
|
||||
GmailMessagesImportService,
|
||||
GmailMessagesImportV2Service,
|
||||
SaveMessagesAndEnqueueContactCreationService,
|
||||
],
|
||||
exports: [GmailMessagesImportService, GmailMessagesImportV2Service],
|
||||
})
|
||||
export class GmailMessagesImportModule {}
|
||||
@ -1,320 +0,0 @@
|
||||
import { Inject, Injectable, Logger } from '@nestjs/common';
|
||||
import { InjectRepository } from '@nestjs/typeorm';
|
||||
|
||||
import { EntityManager, Repository } from 'typeorm';
|
||||
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { FetchMessagesByBatchesService } from 'src/modules/messaging/services/fetch-messages-by-batches/fetch-messages-by-batches.service';
|
||||
import {
|
||||
MessageChannelWorkspaceEntity,
|
||||
MessageChannelSyncStatus,
|
||||
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { createQueriesFromMessageIds } from 'src/modules/messaging/utils/create-queries-from-message-ids.util';
|
||||
import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator';
|
||||
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
|
||||
import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache-storage.service';
|
||||
import { GMAIL_USERS_MESSAGES_GET_BATCH_SIZE } from 'src/modules/messaging/constants/gmail-users-messages-get-batch-size.constant';
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import {
|
||||
GmailFullMessageListFetchJobData,
|
||||
GmailFullMessageListFetchJob,
|
||||
} from 'src/modules/messaging/jobs/gmail-full-message-list-fetch.job';
|
||||
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
|
||||
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
|
||||
import { GMAIL_ONGOING_SYNC_TIMEOUT } from 'src/modules/messaging/constants/gmail-ongoing-sync-timeout.constant';
|
||||
import { MessageParticipantService } from 'src/modules/messaging/services/message-participant/message-participant.service';
|
||||
import { MessageService } from 'src/modules/messaging/services/message/message.service';
|
||||
import { ParticipantWithMessageId } from 'src/modules/messaging/types/gmail-message';
|
||||
import {
|
||||
CreateCompanyAndContactJobData,
|
||||
CreateCompanyAndContactJob,
|
||||
} from 'src/modules/connected-account/auto-companies-and-contacts-creation/jobs/create-company-and-contact.job';
|
||||
import {
|
||||
FeatureFlagEntity,
|
||||
FeatureFlagKeys,
|
||||
} from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
|
||||
@Injectable()
|
||||
export class GmailMessagesImportService {
|
||||
private readonly logger = new Logger(GmailMessagesImportService.name);
|
||||
|
||||
constructor(
|
||||
private readonly fetchMessagesByBatchesService: FetchMessagesByBatchesService,
|
||||
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
|
||||
private readonly connectedAccountRepository: ConnectedAccountRepository,
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
@InjectCacheStorage(CacheStorageNamespace.Messaging)
|
||||
private readonly cacheStorage: CacheStorageService,
|
||||
private readonly workspaceDataSourceService: WorkspaceDataSourceService,
|
||||
@Inject(MessageQueue.messagingQueue)
|
||||
private readonly messageQueueService: MessageQueueService,
|
||||
private readonly messageService: MessageService,
|
||||
private readonly messageParticipantService: MessageParticipantService,
|
||||
@InjectRepository(FeatureFlagEntity, 'core')
|
||||
private readonly featureFlagRepository: Repository<FeatureFlagEntity>,
|
||||
) {}
|
||||
|
||||
async fetchMessageContentFromCache(
|
||||
workspaceId: string,
|
||||
connectedAccountId: string,
|
||||
) {
|
||||
const connectedAccount = await this.connectedAccountRepository.getById(
|
||||
connectedAccountId,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
if (!connectedAccount) {
|
||||
this.logger.error(
|
||||
`Connected account ${connectedAccountId} not found in workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
const { accessToken, refreshToken, authFailedAt } = connectedAccount;
|
||||
|
||||
if (authFailedAt) {
|
||||
this.logger.error(
|
||||
`Connected account ${connectedAccountId} in workspace ${workspaceId} is in a failed state. Skipping...`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
if (!refreshToken) {
|
||||
throw new Error(
|
||||
`No refresh token found for connected account ${connectedAccountId} in workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
|
||||
const gmailMessageChannel =
|
||||
await this.messageChannelRepository.getFirstByConnectedAccountId(
|
||||
connectedAccountId,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
if (!gmailMessageChannel) {
|
||||
this.logger.error(
|
||||
`No message channel found for connected account ${connectedAccountId} in workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
if (gmailMessageChannel.syncStatus !== MessageChannelSyncStatus.PENDING) {
|
||||
this.logger.log(
|
||||
`Messaging import for workspace ${workspaceId} and account ${connectedAccountId} is not pending.`,
|
||||
);
|
||||
|
||||
if (gmailMessageChannel.syncStatus !== MessageChannelSyncStatus.ONGOING) {
|
||||
return;
|
||||
}
|
||||
|
||||
const ongoingSyncStartedAt = new Date(
|
||||
gmailMessageChannel.ongoingSyncStartedAt,
|
||||
);
|
||||
|
||||
if (
|
||||
ongoingSyncStartedAt < new Date(Date.now() - GMAIL_ONGOING_SYNC_TIMEOUT)
|
||||
) {
|
||||
this.logger.log(
|
||||
`Messaging import for workspace ${workspaceId} and account ${connectedAccountId} failed due to ongoing sync timeout. Restarting full-sync...`,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.FAILED,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
await this.fallbackToFullSync(workspaceId, connectedAccountId);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
const gmailMessageChannelId = gmailMessageChannel.id;
|
||||
|
||||
const messageIdsToFetch =
|
||||
(await this.cacheStorage.setPop(
|
||||
`messages-to-import:${workspaceId}:gmail:${gmailMessageChannelId}`,
|
||||
GMAIL_USERS_MESSAGES_GET_BATCH_SIZE,
|
||||
)) ?? [];
|
||||
|
||||
if (!messageIdsToFetch?.length) {
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannelId,
|
||||
MessageChannelSyncStatus.SUCCEEDED,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
this.logger.log(
|
||||
`Messaging import for workspace ${workspaceId} and account ${connectedAccountId} done with nothing to import or delete.`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannelId,
|
||||
MessageChannelSyncStatus.ONGOING,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
this.logger.log(
|
||||
`Messaging import for workspace ${workspaceId} and account ${connectedAccountId} starting...`,
|
||||
);
|
||||
|
||||
const workspaceDataSource =
|
||||
await this.workspaceDataSourceService.connectToWorkspaceDataSource(
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
const messageQueries = createQueriesFromMessageIds(messageIdsToFetch);
|
||||
|
||||
const isContactCreationForSentAndReceivedEmailsEnabledFeatureFlag =
|
||||
await this.featureFlagRepository.findOneBy({
|
||||
workspaceId: workspaceId,
|
||||
key: FeatureFlagKeys.IsContactCreationForSentAndReceivedEmailsEnabled,
|
||||
value: true,
|
||||
});
|
||||
|
||||
const isContactCreationForSentAndReceivedEmailsEnabled =
|
||||
isContactCreationForSentAndReceivedEmailsEnabledFeatureFlag?.value;
|
||||
|
||||
try {
|
||||
const messagesToSave =
|
||||
await this.fetchMessagesByBatchesService.fetchAllMessages(
|
||||
messageQueries,
|
||||
accessToken,
|
||||
workspaceId,
|
||||
connectedAccountId,
|
||||
);
|
||||
|
||||
const participantsWithMessageId = await workspaceDataSource?.transaction(
|
||||
async (transactionManager: EntityManager) => {
|
||||
if (!messagesToSave.length) {
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannelId,
|
||||
MessageChannelSyncStatus.PENDING,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
const messageExternalIdsAndIdsMap =
|
||||
await this.messageService.saveMessagesWithinTransaction(
|
||||
messagesToSave,
|
||||
connectedAccount,
|
||||
gmailMessageChannel.id,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
const participantsWithMessageId: (ParticipantWithMessageId & {
|
||||
shouldCreateContact: boolean;
|
||||
})[] = messagesToSave.flatMap((message) => {
|
||||
const messageId = messageExternalIdsAndIdsMap.get(
|
||||
message.externalId,
|
||||
);
|
||||
|
||||
return messageId
|
||||
? message.participants.map((participant) => ({
|
||||
...participant,
|
||||
messageId,
|
||||
shouldCreateContact:
|
||||
gmailMessageChannel.isContactAutoCreationEnabled &&
|
||||
(isContactCreationForSentAndReceivedEmailsEnabled ||
|
||||
message.participants.find((p) => p.role === 'from')
|
||||
?.handle === connectedAccount.handle),
|
||||
}))
|
||||
: [];
|
||||
});
|
||||
|
||||
await this.messageParticipantService.saveMessageParticipants(
|
||||
participantsWithMessageId,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
if (messageIdsToFetch.length < GMAIL_USERS_MESSAGES_GET_BATCH_SIZE) {
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannelId,
|
||||
MessageChannelSyncStatus.SUCCEEDED,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
this.logger.log(
|
||||
`Messaging import for workspace ${workspaceId} and account ${connectedAccountId} done with no more messages to import.`,
|
||||
);
|
||||
} else {
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannelId,
|
||||
MessageChannelSyncStatus.PENDING,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
this.logger.log(
|
||||
`Messaging import for workspace ${workspaceId} and account ${connectedAccountId} done with more messages to import.`,
|
||||
);
|
||||
}
|
||||
|
||||
return participantsWithMessageId;
|
||||
},
|
||||
);
|
||||
|
||||
if (gmailMessageChannel.isContactAutoCreationEnabled) {
|
||||
const contactsToCreate = participantsWithMessageId.filter(
|
||||
(participant) => participant.shouldCreateContact,
|
||||
);
|
||||
|
||||
await this.messageQueueService.add<CreateCompanyAndContactJobData>(
|
||||
CreateCompanyAndContactJob.name,
|
||||
{
|
||||
workspaceId,
|
||||
connectedAccountHandle: connectedAccount.handle,
|
||||
contactsToCreate,
|
||||
},
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
await this.cacheStorage.setAdd(
|
||||
`messages-to-import:${workspaceId}:gmail:${gmailMessageChannelId}`,
|
||||
messageIdsToFetch,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannelId,
|
||||
MessageChannelSyncStatus.FAILED,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
this.logger.error(
|
||||
`Error fetching messages for ${connectedAccountId} in workspace ${workspaceId}: locking for ${GMAIL_ONGOING_SYNC_TIMEOUT}ms...`,
|
||||
);
|
||||
|
||||
throw new Error(
|
||||
`Error fetching messages for ${connectedAccountId} in workspace ${workspaceId}: ${error.message}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
private async fallbackToFullSync(
|
||||
workspaceId: string,
|
||||
connectedAccountId: string,
|
||||
) {
|
||||
await this.messageQueueService.add<GmailFullMessageListFetchJobData>(
|
||||
GmailFullMessageListFetchJob.name,
|
||||
{ workspaceId, connectedAccountId },
|
||||
);
|
||||
}
|
||||
}
|
||||
@ -1,46 +0,0 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||
|
||||
import { FeatureFlagEntity } from 'src/engine/core-modules/feature-flag/feature-flag.entity';
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { WorkspaceDataSourceModule } from 'src/engine/workspace-datasource/workspace-datasource.module';
|
||||
import { BlocklistWorkspaceEntity } from 'src/modules/connected-account/standard-objects/blocklist.workspace-entity';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import { FetchMessagesByBatchesModule } from 'src/modules/messaging/services/fetch-messages-by-batches/fetch-messages-by-batches.module';
|
||||
import { GmailErrorHandlingModule } from 'src/modules/messaging/services/gmail-error-handling/gmail-error-handling.module';
|
||||
import { GmailFetchMessageIdsToExcludeService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-fetch-messages-ids-to-exclude.service';
|
||||
import { GmailGetHistoryService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-get-history.service';
|
||||
import { GmailPartialMessageListFetchV2Service } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-partial-message-list-fetch-v2.service';
|
||||
import { GmailPartialMessageListFetchService } from 'src/modules/messaging/services/gmail-partial-message-list-fetch/gmail-partial-message-list-fetch.service';
|
||||
import { SetMessageChannelSyncStatusModule } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.module';
|
||||
import { MessageModule } from 'src/modules/messaging/services/message/message.module';
|
||||
import { MessagingProvidersModule } from 'src/modules/messaging/services/providers/messaging-providers.module';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
MessagingProvidersModule,
|
||||
FetchMessagesByBatchesModule,
|
||||
ObjectMetadataRepositoryModule.forFeature([
|
||||
ConnectedAccountWorkspaceEntity,
|
||||
MessageChannelWorkspaceEntity,
|
||||
BlocklistWorkspaceEntity,
|
||||
]),
|
||||
MessageModule,
|
||||
TypeOrmModule.forFeature([FeatureFlagEntity], 'core'),
|
||||
WorkspaceDataSourceModule,
|
||||
SetMessageChannelSyncStatusModule,
|
||||
GmailErrorHandlingModule,
|
||||
],
|
||||
providers: [
|
||||
GmailPartialMessageListFetchService,
|
||||
GmailPartialMessageListFetchV2Service,
|
||||
GmailGetHistoryService,
|
||||
GmailFetchMessageIdsToExcludeService,
|
||||
],
|
||||
exports: [
|
||||
GmailPartialMessageListFetchService,
|
||||
GmailPartialMessageListFetchV2Service,
|
||||
],
|
||||
})
|
||||
export class GmailPartialMessageListFetchModule {}
|
||||
@ -1,347 +0,0 @@
|
||||
import { Inject, Injectable, Logger } from '@nestjs/common';
|
||||
|
||||
import { gmail_v1 } from 'googleapis';
|
||||
import { EntityManager } from 'typeorm';
|
||||
|
||||
import { GmailClientProvider } from 'src/modules/messaging/services/providers/gmail/gmail-client.provider';
|
||||
import { MessageQueueService } from 'src/engine/integrations/message-queue/services/message-queue.service';
|
||||
import { MessageQueue } from 'src/engine/integrations/message-queue/message-queue.constants';
|
||||
import { ConnectedAccountRepository } from 'src/modules/connected-account/repositories/connected-account.repository';
|
||||
import { MessageChannelRepository } from 'src/modules/messaging/repositories/message-channel.repository';
|
||||
import { InjectObjectMetadataRepository } from 'src/engine/object-metadata-repository/object-metadata-repository.decorator';
|
||||
import { ConnectedAccountWorkspaceEntity } from 'src/modules/connected-account/standard-objects/connected-account.workspace-entity';
|
||||
import {
|
||||
MessageChannelWorkspaceEntity,
|
||||
MessageChannelSyncStatus,
|
||||
} from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
import { GMAIL_USERS_HISTORY_MAX_RESULT } from 'src/modules/messaging/constants/gmail-users-history-max-result.constant';
|
||||
import { CacheStorageService } from 'src/engine/integrations/cache-storage/cache-storage.service';
|
||||
import { InjectCacheStorage } from 'src/engine/integrations/cache-storage/decorators/cache-storage.decorator';
|
||||
import { CacheStorageNamespace } from 'src/engine/integrations/cache-storage/types/cache-storage-namespace.enum';
|
||||
import { WorkspaceDataSourceService } from 'src/engine/workspace-datasource/workspace-datasource.service';
|
||||
import {
|
||||
GmailFullMessageListFetchJob,
|
||||
GmailFullMessageListFetchJobData,
|
||||
} from 'src/modules/messaging/jobs/gmail-full-message-list-fetch.job';
|
||||
import { MessageChannelMessageAssociationWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel-message-association.workspace-entity';
|
||||
import { MessageChannelMessageAssociationRepository } from 'src/modules/messaging/repositories/message-channel-message-association.repository';
|
||||
|
||||
@Injectable()
|
||||
export class GmailPartialMessageListFetchService {
|
||||
private readonly logger = new Logger(
|
||||
GmailPartialMessageListFetchService.name,
|
||||
);
|
||||
|
||||
constructor(
|
||||
private readonly gmailClientProvider: GmailClientProvider,
|
||||
@Inject(MessageQueue.messagingQueue)
|
||||
private readonly messageQueueService: MessageQueueService,
|
||||
@InjectObjectMetadataRepository(ConnectedAccountWorkspaceEntity)
|
||||
private readonly connectedAccountRepository: ConnectedAccountRepository,
|
||||
@InjectObjectMetadataRepository(MessageChannelWorkspaceEntity)
|
||||
private readonly messageChannelRepository: MessageChannelRepository,
|
||||
@InjectCacheStorage(CacheStorageNamespace.Messaging)
|
||||
private readonly cacheStorage: CacheStorageService,
|
||||
private readonly workspaceDataSourceService: WorkspaceDataSourceService,
|
||||
@InjectObjectMetadataRepository(
|
||||
MessageChannelMessageAssociationWorkspaceEntity,
|
||||
)
|
||||
private readonly messageChannelMessageAssociationRepository: MessageChannelMessageAssociationRepository,
|
||||
) {}
|
||||
|
||||
public async fetchConnectedAccountThreads(
|
||||
workspaceId: string,
|
||||
connectedAccountId: string,
|
||||
): Promise<void> {
|
||||
const connectedAccount = await this.connectedAccountRepository.getById(
|
||||
connectedAccountId,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
if (!connectedAccount) {
|
||||
this.logger.error(
|
||||
`Connected account ${connectedAccountId} not found in workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
const refreshToken = connectedAccount.refreshToken;
|
||||
|
||||
if (!refreshToken) {
|
||||
throw new Error(
|
||||
`No refresh token found for connected account ${connectedAccountId} in workspace ${workspaceId}`,
|
||||
);
|
||||
}
|
||||
|
||||
const gmailMessageChannel =
|
||||
await this.messageChannelRepository.getFirstByConnectedAccountId(
|
||||
connectedAccountId,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
if (!gmailMessageChannel) {
|
||||
this.logger.error(
|
||||
`No message channel found for connected account ${connectedAccountId} in workspace ${workspaceId}`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
if (gmailMessageChannel.syncStatus !== MessageChannelSyncStatus.SUCCEEDED) {
|
||||
this.logger.log(
|
||||
`Messaging import for workspace ${workspaceId} and account ${connectedAccountId} is locked, import will be retried later.`,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.ONGOING,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
const workspaceDataSource =
|
||||
await this.workspaceDataSourceService.connectToWorkspaceDataSource(
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
await workspaceDataSource
|
||||
?.transaction(async (transactionManager: EntityManager) => {
|
||||
const lastSyncHistoryId = gmailMessageChannel.syncCursor;
|
||||
|
||||
if (!lastSyncHistoryId) {
|
||||
this.logger.log(
|
||||
`No lastSyncHistoryId for workspace ${workspaceId} and account ${connectedAccountId}, falling back to full sync.`,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.PENDING,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
await this.fallbackToFullSync(workspaceId, connectedAccountId);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
const gmailClient: gmail_v1.Gmail =
|
||||
await this.gmailClientProvider.getGmailClient(refreshToken);
|
||||
|
||||
const { history, historyId, error } = await this.getHistoryFromGmail(
|
||||
gmailClient,
|
||||
lastSyncHistoryId,
|
||||
);
|
||||
|
||||
if (error?.code === 404) {
|
||||
this.logger.log(
|
||||
`404: Invalid lastSyncHistoryId: ${lastSyncHistoryId} for workspace ${workspaceId} and account ${connectedAccountId}, falling back to full sync.`,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.resetSyncCursor(
|
||||
gmailMessageChannel.id,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.PENDING,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
await this.fallbackToFullSync(workspaceId, connectedAccountId);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
if (error?.code === 429) {
|
||||
this.logger.log(
|
||||
`429: rate limit reached for workspace ${workspaceId} and account ${connectedAccountId}: ${error.message}, import will be retried later.`,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.PENDING,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
if (error) {
|
||||
throw new Error(
|
||||
`Error fetching messages for ${connectedAccountId} in workspace ${workspaceId}: ${error.message}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (!historyId) {
|
||||
throw new Error(
|
||||
`No historyId found for ${connectedAccountId} in workspace ${workspaceId} in gmail history response.`,
|
||||
);
|
||||
}
|
||||
|
||||
if (historyId === lastSyncHistoryId || !history?.length) {
|
||||
this.logger.log(
|
||||
`Messaging import done with history ${historyId} and nothing to update for workspace ${workspaceId} and account ${connectedAccountId}`,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.PENDING,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
const { messagesAdded, messagesDeleted } =
|
||||
await this.getMessageIdsFromHistory(history);
|
||||
|
||||
await this.cacheStorage.setAdd(
|
||||
`messages-to-import:${workspaceId}:gmail:${gmailMessageChannel.id}`,
|
||||
messagesAdded,
|
||||
);
|
||||
|
||||
await this.messageChannelMessageAssociationRepository.deleteByMessageExternalIdsAndMessageChannelId(
|
||||
messagesDeleted,
|
||||
gmailMessageChannel.id,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.updateLastSyncCursorIfHigher(
|
||||
gmailMessageChannel.id,
|
||||
historyId,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.PENDING,
|
||||
workspaceId,
|
||||
transactionManager,
|
||||
);
|
||||
})
|
||||
.catch(async (error) => {
|
||||
await this.messageChannelRepository.updateSyncStatus(
|
||||
gmailMessageChannel.id,
|
||||
MessageChannelSyncStatus.FAILED,
|
||||
workspaceId,
|
||||
);
|
||||
|
||||
throw new Error(
|
||||
`Error fetching messages for ${connectedAccountId} in workspace ${workspaceId}: ${error.message}`,
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
private async getMessageIdsFromHistory(
|
||||
history: gmail_v1.Schema$History[],
|
||||
): Promise<{
|
||||
messagesAdded: string[];
|
||||
messagesDeleted: string[];
|
||||
}> {
|
||||
const { messagesAdded, messagesDeleted } = history.reduce(
|
||||
(
|
||||
acc: {
|
||||
messagesAdded: string[];
|
||||
messagesDeleted: string[];
|
||||
},
|
||||
history,
|
||||
) => {
|
||||
const messagesAdded = history.messagesAdded?.map(
|
||||
(messageAdded) => messageAdded.message?.id || '',
|
||||
);
|
||||
|
||||
const messagesDeleted = history.messagesDeleted?.map(
|
||||
(messageDeleted) => messageDeleted.message?.id || '',
|
||||
);
|
||||
|
||||
if (messagesAdded) acc.messagesAdded.push(...messagesAdded);
|
||||
if (messagesDeleted) acc.messagesDeleted.push(...messagesDeleted);
|
||||
|
||||
return acc;
|
||||
},
|
||||
{ messagesAdded: [], messagesDeleted: [] },
|
||||
);
|
||||
|
||||
const uniqueMessagesAdded = messagesAdded.filter(
|
||||
(messageId) => !messagesDeleted.includes(messageId),
|
||||
);
|
||||
|
||||
const uniqueMessagesDeleted = messagesDeleted.filter(
|
||||
(messageId) => !messagesAdded.includes(messageId),
|
||||
);
|
||||
|
||||
return {
|
||||
messagesAdded: uniqueMessagesAdded,
|
||||
messagesDeleted: uniqueMessagesDeleted,
|
||||
};
|
||||
}
|
||||
|
||||
private async getHistoryFromGmail(
|
||||
gmailClient: gmail_v1.Gmail,
|
||||
lastSyncHistoryId: string,
|
||||
): Promise<{
|
||||
history: gmail_v1.Schema$History[];
|
||||
historyId?: string | null;
|
||||
error?: any;
|
||||
}> {
|
||||
const fullHistory: gmail_v1.Schema$History[] = [];
|
||||
let pageToken: string | undefined;
|
||||
let hasMoreMessages = true;
|
||||
let nextHistoryId: string | undefined;
|
||||
|
||||
while (hasMoreMessages) {
|
||||
try {
|
||||
const response = await gmailClient.users.history.list({
|
||||
userId: 'me',
|
||||
maxResults: GMAIL_USERS_HISTORY_MAX_RESULT,
|
||||
pageToken,
|
||||
startHistoryId: lastSyncHistoryId,
|
||||
historyTypes: ['messageAdded', 'messageDeleted'],
|
||||
});
|
||||
|
||||
nextHistoryId = response?.data?.historyId ?? undefined;
|
||||
|
||||
if (response?.data?.history) {
|
||||
fullHistory.push(...response.data.history);
|
||||
}
|
||||
|
||||
pageToken = response?.data?.nextPageToken ?? undefined;
|
||||
hasMoreMessages = !!pageToken;
|
||||
} catch (error) {
|
||||
const errorData = error?.response?.data?.error;
|
||||
|
||||
if (errorData) {
|
||||
return {
|
||||
history: [],
|
||||
error: errorData,
|
||||
historyId: lastSyncHistoryId,
|
||||
};
|
||||
}
|
||||
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
return { history: fullHistory, historyId: nextHistoryId };
|
||||
}
|
||||
|
||||
private async fallbackToFullSync(
|
||||
workspaceId: string,
|
||||
connectedAccountId: string,
|
||||
) {
|
||||
await this.messageQueueService.add<GmailFullMessageListFetchJobData>(
|
||||
GmailFullMessageListFetchJob.name,
|
||||
{ workspaceId, connectedAccountId },
|
||||
);
|
||||
}
|
||||
}
|
||||
@ -1,14 +0,0 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
|
||||
import { ObjectMetadataRepositoryModule } from 'src/engine/object-metadata-repository/object-metadata-repository.module';
|
||||
import { MessageChannelSyncStatusService } from 'src/modules/messaging/services/message-channel-sync-status/message-channel-sync-status.service';
|
||||
import { MessageChannelWorkspaceEntity } from 'src/modules/messaging/standard-objects/message-channel.workspace-entity';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
ObjectMetadataRepositoryModule.forFeature([MessageChannelWorkspaceEntity]),
|
||||
],
|
||||
providers: [MessageChannelSyncStatusService],
|
||||
exports: [MessageChannelSyncStatusService],
|
||||
})
|
||||
export class SetMessageChannelSyncStatusModule {}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user