This PR introduces a significant enhancement to the role-based
permission system by extending it to support AI agents, enabling them to
perform database operations based on assigned permissions.
## Key Changes
### 1. Database Schema Migration
- **Table Rename**: `userWorkspaceRole` → `roleTargets` to better
reflect its expanded purpose
- **New Column**: Added `agentId` (UUID, nullable) to support AI agent
role assignments
- **Constraint Updates**:
- Made `userWorkspaceId` nullable to accommodate agent-only role
assignments
- Added check constraint `CHK_role_targets_either_agent_or_user`
ensuring either `agentId` OR `userWorkspaceId` is set (not both)
### 2. Entity & Service Layer Updates
- **RoleTargetsEntity**: Updated with new `agentId` field and constraint
validation
- **AgentRoleService**: New service for managing agent role assignments
with validation
- **AgentService**: Enhanced to include role information when retrieving
agents
- **RoleResolver**: Added GraphQL mutations for `assignRoleToAgent` and
`removeRoleFromAgent`
### 3. AI Agent CRUD Operations
- **Permission-Based Tool Generation**: AI agents now receive database
tools based on their assigned role permissions
- **Dynamic Tool Creation**: The `AgentToolService` generates CRUD tools
(`create_*`, `find_*`, `update_*`, `soft_delete_*`, `destroy_*`) for
each object based on role permissions
- **Granular Permissions**: Supports both global role permissions
(`canReadAllObjectRecords`) and object-specific permissions
(`canReadObjectRecords`)
### 4. Frontend Integration
- **Role Assignment UI**: Added hooks and components for
assigning/removing roles from agents
## Demo
https://github.com/user-attachments/assets/41732267-742e-416c-b423-b687c2614c82
---------
Co-authored-by: Antoine Moreaux <moreaux.antoine@gmail.com>
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: Guillim <guillim@users.noreply.github.com>
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@twenty.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Marie <51697796+ijreilly@users.noreply.github.com>
Co-authored-by: martmull <martmull@hotmail.fr>
Co-authored-by: Thomas Trompette <thomas.trompette@sfr.fr>
Co-authored-by: Etienne <45695613+etiennejouan@users.noreply.github.com>
Co-authored-by: Baptiste Devessier <baptiste@devessier.fr>
Co-authored-by: nitin <142569587+ehconitin@users.noreply.github.com>
Co-authored-by: Paul Rastoin <45004772+prastoin@users.noreply.github.com>
Co-authored-by: prastoin <paul@twenty.com>
Co-authored-by: Vicky Wang <157669812+vickywxng@users.noreply.github.com>
Co-authored-by: Vicky Wang <vw92@cornell.edu>
Co-authored-by: Raphaël Bosi <71827178+bosiraphael@users.noreply.github.com>
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
This PR implements a global PostgreSQL connection pool sharing
mechanism.
- Patches pg.Pool to reuse connection pools across the application when
connection parameters match, reducing resource overhead.
- New environment variables allow enabling/disabling sharing and
configuring pool size, idle timeout, and client exit behavior.
WorkspaceDatasourceFactory will now use shared pools if enabled, this
will avoid recreating 10 connections for each pods for each workspace.
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
In this PR we are
- introducing a cached map `{ userworkspaceId: roleId } `to reduce calls
to get a userWorkspace's role (we were having N+1 around that with
combinedFindMany queries and generally having a lot of avoidable
queries)
- using the roles permissions cache (`{ roleId: { objectNameSingular:
{ canRead: bool, canUpdate: bool, ...} } `) in Permissions V1's
userHasObjectPermission, in order to 1) improve performances to avoid
calls to get roles 2) start using our permissions cache
In this PR we are
- (if permissionsV2 is enabled) executing permission checks at query
builder level. To do so we want to override the query builders methods
that are performing db calls (.execute(), .getMany(), ... etc.) For now
I have just overriden some of the query builders methods for the poc. To
do so I created custom query builder classes that extend typeorm's query
builder (selectQueryBuilder and updateQueryBuilder, for now and later I
will tackle softDeleteQueryBuilder, etc.).
- adding a notion of roles permissions version and roles permissions
object to datasources. We will now use one datasource per roleId and
rolePermissionVersion. Both rolesPermissionsVersion and rolesPermissions
objects are stored in redis and recomputed at role update or if queried
and found empty. Unlike for metadata version we don't need to store a
version in the db that stands for the source of truth. We also don't
need to destroy and recreate the datasource if the rolesPermissions
version changes, but only to update the value for rolesPermissions and
rolesPermissionsVersions on the existing datasource.
What this PR misses
- computing of roles permissions should take into account
objectPermissions table (for now it only looks at what's on the roles
table)
- pursue extension of query builder classes and overriding of their db
calling-methods
- what should the behaviour be for calls from twentyOrmGlobalManager
that don't have a roleId?
This PR introduces the following changes:
- add the metadataVersion to all our metadata cache keys to ease
troubleshooting:
<img width="1146" alt="image"
src="https://github.com/user-attachments/assets/8427805b-e07f-465e-9e69-1403652c8b12">
- introduce a cache recompute lock to avoid overloading the database to
recompute the cache many time
## Context
The goal is to replace pg_graphql with our own ORM wrapper (TwentyORM).
This PR tries to add some parsing logic to convert graphql requests to
send to the ORM to replace pg_graphql implementation.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
### Overview
This PR introduces significant enhancements to the MessageQueue module
by integrating `@Processor`, `@Process`, and `@InjectMessageQueue`
decorators. These changes streamline the process of defining and
managing queue processors and job handlers, and also allow for
request-scoped handlers, improving compatibility with services that rely
on scoped providers like TwentyORM repositories.
### Key Features
1. **Decorator-based Job Handling**: Use `@Processor` and `@Process`
decorators to define job handlers declaratively.
2. **Request Scope Support**: Job handlers can be scoped per request,
enhancing integration with request-scoped services.
### Usage
#### Defining Processors and Job Handlers
The `@Processor` decorator is used to define a class that processes jobs
for a specific queue. The `@Process` decorator is applied to methods
within this class to define specific job handlers.
##### Example 1: Specific Job Handlers
```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';
@Processor('taskQueue')
export class TaskProcessor {
@Process('taskA')
async handleTaskA(job: { id: string, data: any }) {
console.log(`Handling task A with data:`, job.data);
// Logic for task A
}
@Process('taskB')
async handleTaskB(job: { id: string, data: any }) {
console.log(`Handling task B with data:`, job.data);
// Logic for task B
}
}
```
In the example above, `TaskProcessor` is responsible for processing jobs
in the `taskQueue`. The `handleTaskA` method will only be called for
jobs with the name `taskA`, while `handleTaskB` will be called for
`taskB` jobs.
##### Example 2: General Job Handler
```typescript
import { Processor, Process, InjectMessageQueue } from 'src/engine/integrations/message-queue';
@Processor('generalQueue')
export class GeneralProcessor {
@Process()
async handleAnyJob(job: { id: string, name: string, data: any }) {
console.log(`Handling job ${job.name} with data:`, job.data);
// Logic for any job
}
}
```
In this example, `GeneralProcessor` handles all jobs in the
`generalQueue`, regardless of the job name. The `handleAnyJob` method
will be invoked for every job added to the `generalQueue`.
#### Adding Jobs to a Queue
You can use the `@InjectMessageQueue` decorator to inject a queue into a
service and add jobs to it.
##### Example:
```typescript
import { Injectable } from '@nestjs/common';
import { InjectMessageQueue, MessageQueue } from 'src/engine/integrations/message-queue';
@Injectable()
export class TaskService {
constructor(
@InjectMessageQueue('taskQueue') private readonly taskQueue: MessageQueue,
) {}
async addTaskA(data: any) {
await this.taskQueue.add('taskA', data);
}
async addTaskB(data: any) {
await this.taskQueue.add('taskB', data);
}
}
```
In this example, `TaskService` adds jobs to the `taskQueue`. The
`addTaskA` and `addTaskB` methods add jobs named `taskA` and `taskB`,
respectively, to the queue.
#### Using Scoped Job Handlers
To utilize request-scoped job handlers, specify the scope in the
`@Processor` decorator. This is particularly useful for services that
use scoped repositories like those in TwentyORM.
##### Example:
```typescript
import { Processor, Process, InjectMessageQueue, Scope } from 'src/engine/integrations/message-queue';
@Processor({ name: 'scopedQueue', scope: Scope.REQUEST })
export class ScopedTaskProcessor {
@Process('scopedTask')
async handleScopedTask(job: { id: string, data: any }) {
console.log(`Handling scoped task with data:`, job.data);
// Logic for scoped task, which might use request-scoped services
}
}
```
Here, the `ScopedTaskProcessor` is associated with `scopedQueue` and
operates with request scope. This setup is essential when the job
handler relies on services that need to be instantiated per request,
such as scoped repositories.
### Migration Notes
- **Decorators**: Refactor job handlers to use `@Processor` and
`@Process` decorators.
- **Request Scope**: Utilize the scope option in `@Processor` if your
job handlers depend on request-scoped services.
Fix#5628
---------
Co-authored-by: Weiko <corentin@twenty.com>
## Introduction
This PR introduces "TwentyORM," a custom ORM module designed to
streamline database interactions within our workspace schema, reducing
the need for raw SQL queries. The API mirrors TypeORM's to provide a
familiar interface while integrating enhancements specific to our
project's needs.
To facilitate this integration, new decorators prefixed with `Workspace`
have been implemented. These decorators are used to define entity
metadata more explicitly and are critical in constructing our schema
dynamically.
## New Features
- **Custom ORM System**: Named "TwentyORM," which aligns closely with
TypeORM for ease of use but is tailored to our application's specific
requirements.
- **Decorator-Driven Configuration**: Entities are now configured with
`Workspace`-prefixed decorators that clearly define schema mappings and
relationships directly within the entity classes.
- **Injectable Repositories**: Repositories can be injected similarly to
TypeORM, allowing for flexible and straightforward data management.
## Example Implementations
### Decorated Entity Definitions
Entities are defined with new decorators that outline table and field
metadata, relationships, and constraints. Here are examples of these
implementations:
#### Company Metadata Object
```typescript
@WorkspaceObject({
standardId: STANDARD_OBJECT_IDS.company,
namePlural: 'companies',
labelSingular: 'Company',
labelPlural: 'Companies',
description: 'A company',
icon: 'IconBuildingSkyscraper',
})
export class CompanyObjectMetadata extends BaseObjectMetadata {
@WorkspaceField({
standardId: COMPANY_STANDARD_FIELD_IDS.name,
type: FieldMetadataType.TEXT,
label: 'Name',
description: 'The company name',
icon: 'IconBuildingSkyscraper',
})
name: string;
@WorkspaceField({
standardId: COMPANY_STANDARD_FIELD_IDS.xLink,
type: FieldMetadataType.LINK,
label: 'X',
description: 'The company Twitter/X account',
icon: 'IconBrandX',
})
@WorkspaceIsNullable()
xLink: LinkMetadata;
@WorkspaceField({
standardId: COMPANY_STANDARD_FIELD_IDS.position,
type: FieldMetadataType.POSITION,
label: 'Position',
description: 'Company record position',
icon: 'IconHierarchy2',
})
@WorkspaceIsSystem()
@WorkspaceIsNullable()
position: number;
@WorkspaceRelation({
standardId: COMPANY_STANDARD_FIELD_IDS.accountOwner,
label: 'Account Owner',
description: 'Your team member responsible for managing the company account',
type: RelationMetadataType.MANY_TO_ONE,
inverseSideTarget: () => WorkspaceMemberObjectMetadata,
inverseSideFieldKey: 'accountOwnerForCompanies',
onDelete: RelationOnDeleteAction.SET_NULL,
})
@WorkspaceIsNullable()
accountOwner: WorkspaceMemberObjectMetadata;
}
```
#### Workspace Member Metadata Object
```typescript
@WorkspaceObject({
standardId: STANDARD_OBJECT_IDS.workspaceMember,
namePlural: 'workspaceMembers',
labelSingular: 'Workspace Member',
labelPlural: 'Workspace Members',
description: 'A workspace member',
icon: 'IconUserCircle',
})
@WorkspaceIsSystem()
@WorkspaceIsNotAuditLogged()
export class WorkspaceMemberObjectMetadata extends BaseObjectMetadata {
@WorkspaceField({
standardId: WORKSPACE_MEMBER_STANDARD_FIELD_IDS.name,
type: FieldMetadataType.FULL_NAME,
label: 'Name',
description: 'Workspace member name',
icon: 'IconCircleUser',
})
name: FullNameMetadata;
@WorkspaceRelation({
standardId: WORKSPACE_MEMBER_STANDARD_FIELD_IDS.accountOwnerForCompanies,
label: 'Account Owner For Companies',
description: 'Account owner for companies',
icon: 'IconBriefcase',
type: RelationMetadataType.ONE_TO_MANY,
inverseSideTarget: () => CompanyObjectMetadata,
inverseSideFieldKey: 'accountOwner',
onDelete: RelationOnDeleteAction.SET_NULL,
})
accountOwnerForCompanies: Relation
<CompanyObjectMetadata[]>;
}
```
### Injectable Repository Usage
Repositories can be directly injected into services, allowing for
streamlined query operations:
```typescript
export class CompanyService {
constructor(
@InjectWorkspaceRepository(CompanyObjectMetadata)
private readonly companyObjectMetadataRepository: WorkspaceRepository<CompanyObjectMetadata>,
) {}
async companies(): Promise<CompanyObjectMetadata[]> {
// Example queries demonstrating simple and relation-loaded operations
const simpleCompanies = await this.companyObjectMetadataRepository.find({});
const companiesWithOwners = await this.companyObjectMetadataRepository.find({
relations: ['accountOwner'],
});
const companiesFilteredByLinkLabel = await this.companyObjectMetadataRepository.find({
where: { xLinkLabel: 'MyLabel' },
});
return companiesFilteredByLinkLabel;
}
}
```
## Conclusions
This PR sets the foundation for a decorator-driven ORM layer that
simplifies data interactions and supports complex entity relationships
while maintaining clean and manageable code architecture. This is not
finished yet, and should be extended.
All the standard objects needs to be migrated and all the module using
the old decorators too.
---------
Co-authored-by: Weiko <corentin@twenty.com>
