Commit Graph

142 Commits

Author SHA1 Message Date
c8a889995f Add error handling service for calendar import (#6203)
Add error handling service for calendar import

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-07-12 18:56:45 +02:00
52aa9abd73 Remove old message channel sync statuses and create migration command (#6177)
Remove old message channel sync statuses and create migration command

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-07-12 18:36:42 +02:00
faf462ffe4 Fix address field in raw query (#6226)
The old address field has been deprecated but the raw query in
company.repository hasn't been updated accordingly
2024-07-11 19:48:15 +02:00
5ebde33f5f Deprecate Probability field on Opportunity (#6207)
Closes #5735.
The field probability on opportunity will -
- stop being created for new workspaces (after this PR is merged)
- have "isCustom" value set to true and be displayed as such in the
settings (after this PR is merged + sync-metadata is run on workspace)
- still show in the views (all the time)

This field is deprecated as a standard field but not replaced by another
one, so we are not adding the `(deprecated)` suffix in the label.
2024-07-11 14:50:33 +02:00
8e25a107fd Add new Address field to views containing deprecated address (#6205)
as per title, following introduction of new Address field, we want to
display the new field next to the deprecated field, for users to notice
the new field.

<img width="983" alt="Capture d’écran 2024-07-10 à 17 44 25"
src="https://github.com/twentyhq/twenty/assets/51697796/7b5309b4-b22d-4f32-8054-68bc7b0f3bb3">
2024-07-11 14:39:38 +02:00
34d13a7b58 Deprecate address standard field (#6087)
Closes #5916

---------

Co-authored-by: Weiko <corentin@twenty.com>
2024-07-10 18:07:18 +02:00
b14918c4b5 Remove featureFlag on connectedAccount.handleAliases (#6202)
Removing the gating on connectedAccount handleAlias as the feature has
been tested for a week
2024-07-10 14:49:22 +02:00
28387003d2 Fix contact creation and rename email aliases to handle aliases (#6176)
Fix contact creation (linked to #6162) and rename email aliases to
handle aliases
2024-07-09 17:49:03 +02:00
37fb88c533 Modify messaging message channel sync status monitoring cron pattern (#6173)
Change from every hour to every 10 minutes, starting at 2 minutes past
the hour
2024-07-09 14:42:57 +02:00
3249c52767 Add missing objectMetadataId column in auditLog (#6164)
Insert inside AuditLog table are all failing due to objectMetadataId
column missing.
The FieldMetadata was sharing the same standard-id with another one
(objectName) so it was skipped during the comparison step of the
sync-metadata.

Running a sync-metadata again should fix this issue. Note that this
column is non-nullable so if the table contains existing records, it
will fail. However, since the insert was failing I'm assuming the table
is empty anyway.
2024-07-09 13:26:09 +02:00
3e453054b7 Fix contactCreation ignoring connectedAccount mainHandle 2024-07-08 18:13:10 +02:00
5638af4385 Fix wrong email direction (#6163)
Closes #6162
2024-07-08 17:55:49 +02:00
f458322303 Refactor calendar to use new sync statuses and stages (#6141)
- Refactor calendar modules and some messaging modules to better
organize them by business rules and decouple them
- Work toward a common architecture for the different calendar providers
by introducing interfaces for the drivers
- Modify cron job to use the new sync statuses and stages
2024-07-08 17:01:06 +02:00
9ba211055a Add message import granulary on non-pro emails, group emails and received contact creation (#6156)
1) Remove featureFlag
2) Base contactCreation on messageChannel.autoContactCreationPolicy
4) add excludeProfessionalEmails + excludeGroupEmails logic
2024-07-08 14:33:48 +02:00
b33b468679 Add command to update boolean fields null values (#6113)
We have recently decided that boolean fields should only accept truthy
or falsy value, with users deciding of a default value at creation.

This command helps cleaning the existing data, by
1. updating all boolean fields default values from null to false
2. updating all boolean fields values for records from null to false

---------

Co-authored-by: Weiko <corentin@twenty.com>
2024-07-05 12:27:12 +02:00
14cd6f8b6b Decrease messaging import batch size 2024-07-03 22:24:21 +02:00
4c57e838ca Fix pg-boss worker not working with dynamic injection (#6119) 2024-07-03 19:17:29 +02:00
a163ccced6 Fix calendar import cron job (#6096)
An error was introduced in the calendar cron job because we tried to
inject the workspace context inside the calendarChannelRepository where
we didn't have access to that context.
2024-07-02 16:51:10 +02:00
f8dd2cc733 Reorganise calendar module (#6089)
Refactor Calendar into functional sub modules
<img width="437" alt="image"
src="https://github.com/twentyhq/twenty/assets/12035771/d9de3285-a226-4fe8-b3ef-2d8a21def2a5">

---------

Co-authored-by: bosiraphael <raphael.bosi@gmail.com>
2024-07-02 13:55:11 +02:00
b371c77284 Change Messaging import frequency 2024-07-01 14:22:31 +02:00
8c33d91734 5748 Create contacts for emails sent and received by email aliases (#5855)
Closes #5748
- Create feature flag
- Add scope `https://www.googleapis.com/auth/profile.emails.read` when
connecting an account
- Get email aliases with google people API, store them in
connectedAccount and refresh them before each message-import
- Update the contact creation logic accordingly
- Refactor

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-07-01 14:21:34 +02:00
632e34fc7b fix: message cleaner find operator (#6080)
This PR fix an issue with the `IsNull()` find operator applied on
one-to-many relation, this one is not supported by TypeORM.
We can instead filter by an empty array to retrieve object with empty
relations.
2024-07-01 11:23:22 +02:00
13f213a05e feat: message cleaner drop repository (#6052)
This PR use the new `TwentyORM` for the message-cleaner module by using
the new injection system with `@InjectWorkspaceRepository`.
2024-06-30 21:57:18 +02:00
b8f33f6f59 5095 move onboardingstatus computation from frontend to backend (#5954)
- move front `onboardingStatus` computing to server side
- add logic to `useSetNextOnboardingStatus`
- update some missing redirections in
`usePageChangeEffectNavigateLocation`
- separate subscriptionStatus from onboardingStatus
2024-06-28 17:32:02 +02:00
1a66db5bff Refactor messaging refresh access token (#6034)
- Put error handling outside of `refreshAndSaveAccessToken`
- return after failing to refresh access token in
`processMessageBatchImport`
- remove unnecessary token refresh in `processMessageListFetch`
2024-06-27 17:07:45 +02:00
4f9527c860 5901 refactor email and calendar auto contact creation to create them by batch (#6038)
Closes #5901
2024-06-27 16:37:34 +02:00
d1bb0fb822 Change messaging batch size and cron pattern (#6063)
Change messaging batch size and cron pattern to accelerate messages
import
2024-06-27 16:23:47 +02:00
ac7d905c84 Create fields for calendar and messaging settings v2 (#6049)
Create fields for calendar and messaging settings v2
2024-06-27 16:23:25 +02:00
845fcb65ce Replace ObjectRecord<MessageChannelWorkspaceEntity> with MessageChannelWorkspaceEntity (#6059)
Replace ObjectRecord<MessageChannelWorkspaceEntity> with
MessageChannelWorkspaceEntity
2024-06-27 14:35:25 +02:00
95c5602a4e feat: manually implement joinColumn (#6022)
This PR introduce a new decorator named `@WorkspaceJoinColumn`, the goal
of this one is to manually declare the join columns inside the workspace
entities, so we don't have to rely on `ObjectRecord` type.

This decorator can be used that way:

```typescript
  @WorkspaceRelation({
    standardId: ACTIVITY_TARGET_STANDARD_FIELD_IDS.company,
    type: RelationMetadataType.MANY_TO_ONE,
    label: 'Company',
    description: 'ActivityTarget company',
    icon: 'IconBuildingSkyscraper',
    inverseSideTarget: () => CompanyWorkspaceEntity,
    inverseSideFieldKey: 'activityTargets',
  })
  @WorkspaceIsNullable()
  company: Relation<CompanyWorkspaceEntity> | null;

  // The argument is the name of the relation above
  @WorkspaceJoinColumn('company')
  companyId: string | null;
```
2024-06-27 11:41:22 +02:00
97822533db Fix cache flush in messaging-channel-sync-status.service (#6024)
Fix cache flush in messaging-channel-sync-status.service
2024-06-27 09:58:37 +02:00
03b00c49d5 Improve gmail error handling by catching and throttling for 400 failedPrecondition (#6044)
Closes #5897
2024-06-27 09:57:19 +02:00
cf67ed09d0 Upsert endpoint and CSV import upsert (#5970)
This PR introduces an `upsert` parameter (along the existing `data`
param) for `createOne` and `createMany` mutations.

When upsert is set to `true`, the function will look for records with
the same id if an id was passed. If not id was passed, it will leverage
the existing duplicate check mechanism to find a duplicate. If a record
is found, then the function will perform an update instead of a create.

Unfortunately I had to remove some nice tests that existing on the args
factory. Those tests where mostly testing the duplication rule
generation logic but through a GraphQL angle. Since I moved the
duplication rule logic to a dedicated service, if I kept the tests but
mocked the service we wouldn't really be testing anything useful. The
right path would be to create new tests for this service that compare
the JSON output and not the GraphQL output but I chose not to work on
this as it's equivalent to rewriting the tests from scratch and I have
other competing priorities.
2024-06-26 11:39:16 +02:00
1736aee7ff Remove message-import cache when connectedAccount is removed (#6021) 2024-06-26 11:23:08 +02:00
7c2e745b45 feat: Dynamic hook registration for WorkspaceQueryHooks (#6008)
#### Overview

This PR introduces a new API for dynamically registering and executing
pre and post query hooks in the Workspace Query Hook system using the
`@WorkspaceQueryHook` decorator. This approach eliminates the need for
manual provider registration, and fix the issue of `undefined` or `null`
repository using `@InjectWorkspaceRepository`.

#### New API

**Define a Hook**

Use the `@WorkspaceQueryHook` decorator to define pre or post hooks:

```typescript
@WorkspaceQueryHook({
  key: `calendarEvent.findMany`,
  scope: Scope.REQUEST,
})
export class CalendarEventFindManyPreQueryHook implements WorkspaceQueryHookInstance {
  async execute(userId: string, workspaceId: string, payload: FindManyResolverArgs): Promise<void> {
    if (!payload?.filter?.id?.eq) {
      throw new BadRequestException('id filter is required');
    }

    // Implement hook logic here
  }
}
```

This API simplifies the registration and execution of query hooks,
providing a more flexible and maintainable approach.

---------

Co-authored-by: Weiko <corentin@twenty.com>
2024-06-25 12:41:46 +02:00
4dfca45fd3 5615 create messageongoingstalecron (#6005)
Closes #5615
2024-06-25 11:57:02 +02:00
a001bf1514 5951 create a command to trigger the import of a single message (#5962)
Closes #5951

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-06-24 18:01:22 +02:00
f3701281e9 Create new sync statuses and stages for calendar (#5997)
Create fields:
- syncStatus
- syncStage
- syncStageStartedAt
2024-06-24 16:39:56 +02:00
0b4bfce324 feat: drop calendar repository (#5824)
This PR is replacing and removing all the raw queries and repositories
with the new `TwentyORM` and injection system using
`@InjectWorkspaceRepository`.
Some logic that was contained inside repositories has been moved to the
services.
In this PR we're only replacing repositories for calendar feature.

---------

Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: bosiraphael <raphael.bosi@gmail.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
2024-06-22 09:26:58 +02:00
86f95c0870 5898 Create a cron to monitor messageChannelSyncStatus (#5933)
Closes #5898
2024-06-19 16:04:01 +02:00
016132ecf6 Fix reconnect google account bug (#5905)
Update syncStage to FULL_MESSAGE_LIST_FETCH_PENDING when reconnecting
the account to trigger a full sync on the next cron iteration.
2024-06-19 16:00:39 +02:00
d99b9d1d6b feat: Enhancements to MessageQueue Module with Decorators (#5657)
### Overview

This PR introduces significant enhancements to the MessageQueue module
by integrating `@Processor`, `@Process`, and `@InjectMessageQueue`
decorators. These changes streamline the process of defining and
managing queue processors and job handlers, and also allow for
request-scoped handlers, improving compatibility with services that rely
on scoped providers like TwentyORM repositories.

### Key Features

1. **Decorator-based Job Handling**: Use `@Processor` and `@Process`
decorators to define job handlers declaratively.
2. **Request Scope Support**: Job handlers can be scoped per request,
enhancing integration with request-scoped services.

### Usage

#### Defining Processors and Job Handlers

The `@Processor` decorator is used to define a class that processes jobs
for a specific queue. The `@Process` decorator is applied to methods
within this class to define specific job handlers.

##### Example 1: Specific Job Handlers

```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';

@Processor('taskQueue')
export class TaskProcessor {

  @Process('taskA')
  async handleTaskA(job: { id: string, data: any }) {
    console.log(`Handling task A with data:`, job.data);
    // Logic for task A
  }

  @Process('taskB')
  async handleTaskB(job: { id: string, data: any }) {
    console.log(`Handling task B with data:`, job.data);
    // Logic for task B
  }
}
```

In the example above, `TaskProcessor` is responsible for processing jobs
in the `taskQueue`. The `handleTaskA` method will only be called for
jobs with the name `taskA`, while `handleTaskB` will be called for
`taskB` jobs.

##### Example 2: General Job Handler

```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';

@Processor('generalQueue')
export class GeneralProcessor {

  @Process()
  async handleAnyJob(job: { id: string, name: string, data: any }) {
    console.log(`Handling job ${job.name} with data:`, job.data);
    // Logic for any job
  }
}
```

In this example, `GeneralProcessor` handles all jobs in the
`generalQueue`, regardless of the job name. The `handleAnyJob` method
will be invoked for every job added to the `generalQueue`.

#### Adding Jobs to a Queue

You can use the `@InjectMessageQueue` decorator to inject a queue into a
service and add jobs to it.

##### Example:

```typescript
import { Injectable } from '@nestjs/common';
import { InjectMessageQueue, MessageQueue } from 'src/engine/integrations/message-queue';

@Injectable()
export class TaskService {
  constructor(
    @InjectMessageQueue('taskQueue') private readonly taskQueue: MessageQueue,
  ) {}

  async addTaskA(data: any) {
    await this.taskQueue.add('taskA', data);
  }

  async addTaskB(data: any) {
    await this.taskQueue.add('taskB', data);
  }
}
```

In this example, `TaskService` adds jobs to the `taskQueue`. The
`addTaskA` and `addTaskB` methods add jobs named `taskA` and `taskB`,
respectively, to the queue.

#### Using Scoped Job Handlers

To utilize request-scoped job handlers, specify the scope in the
`@Processor` decorator. This is particularly useful for services that
use scoped repositories like those in TwentyORM.

##### Example:

```typescript
import { Processor, Process, InjectMessageQueue, Scope } from 'src/engine/integrations/message-queue';

@Processor({ name: 'scopedQueue', scope: Scope.REQUEST })
export class ScopedTaskProcessor {

  @Process('scopedTask')
  async handleScopedTask(job: { id: string, data: any }) {
    console.log(`Handling scoped task with data:`, job.data);
    // Logic for scoped task, which might use request-scoped services
  }
}
```

Here, the `ScopedTaskProcessor` is associated with `scopedQueue` and
operates with request scope. This setup is essential when the job
handler relies on services that need to be instantiated per request,
such as scoped repositories.

### Migration Notes

- **Decorators**: Refactor job handlers to use `@Processor` and
`@Process` decorators.
- **Request Scope**: Utilize the scope option in `@Processor` if your
job handlers depend on request-scoped services.

Fix #5628

---------

Co-authored-by: Weiko <corentin@twenty.com>
2024-06-17 09:49:37 +02:00
82741d3b04 Fix error log on message import (#5866)
Modify #5863 to log the connected account id rather than the message
channel id to be consistent with the other logs and stringify the error.
2024-06-14 12:38:35 +02:00
85fd801480 Add log for errors on message import (#5863)
As per title :)
2024-06-14 09:36:39 +02:00
81c4939812 Fix timeline activity missing updated fields (#5854)
## Before
<img width="883" alt="Screenshot 2024-06-13 at 14 48 05"
src="https://github.com/twentyhq/twenty/assets/1834158/0e72ead1-2d59-4ee3-af3b-dfdd593b7f2e">


## After
<img width="908" alt="Screenshot 2024-06-13 at 14 48 14"
src="https://github.com/twentyhq/twenty/assets/1834158/00e32ddf-e40d-4cc9-a80a-9f5b76bd377a">
2024-06-13 15:22:15 +02:00
f825bea071 5629 update blocklist for messaging v2 (#5756)
Closes #5629 

- Add subdomain support in blocklist (if @example.com is blocked, every
subdomain will be blocked)
2024-06-13 07:53:28 +02:00
374237a988 Refacto rest api, fix graphl playground, improve analytics (#5844)
- Improve the rest api by introducing startingAfter/endingBefore (we
previously had lastCursor), and moving pageInfo/totalCount outside of
the data object.
- Fix broken GraphQL playground on website
- Improve analytics by sending server url
2024-06-12 21:54:33 +02:00
5ec98b5ac3 Fix not possible to read message from other workspaceMember 2024-06-12 08:21:35 +02:00
be96c68416 POC timeline activity (#5697)
TODO: 
- remove WorkspaceIsNotAuditLogged decorators on activity/activityTarget
to log task/note creations
- handle attachments
-  fix css and remove unnecessary styled components or duplicates
2024-06-11 18:53:28 +02:00
64b8e4ec4d Fix access token refresh (#5825)
In `messaging-gmail-messages-import.service`, we were refreshing the
access token before each query but we were passing the old access token
to `fetchAllMessages`.
I modified the function to query the updated connectedAccount with the
new access token.
This will solve the 401 errors we were getting in production.
2024-06-11 18:52:38 +02:00