Commit Graph

4 Commits

Author SHA1 Message Date
d5c974054d Improve performance on metadata computation (#12785)
In this PR:

## Improve recompute metadata cache performance. We are aiming for
~100ms

Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing

## Remove caching lock

As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces

## Introduce self recovery mecanisms to recompute cache automatically if
corrupted

Aka getFreshObjectMetadataMaps

## custom object resolver performance improvement:  1sec to 200ms

Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap

## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>

We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge

## Add dataloader on IndexMetadata (aka indexMetadataList in the API)

## Improve field resolver performances too

## Deprecate ClientConfig
2025-06-23 21:06:17 +02:00
0c60fa9c23 Fix cacheData not found after recomputes (#12032)
Closes https://github.com/twentyhq/core-team-issues/issues/861

sentries: [User workspace role map not found after
recompute](https://twenty-v7.sentry.io/issues/6575092700/events/f9825338a30b470eb2345fe78c1e3479/?project=4507072499810304&query=is%3Aunresolved%20issue.priority%3A%5Bhigh%2C%20medium%5D%20not%20found%20after%20recompute&referrer=next-event&stream_index=0)
(64 events in 90d), [Feature flag map not found after
recompute](https://twenty-v7.sentry.io/issues/6547696076/?project=4507072499810304&query=is%3Aunresolved%20issue.priority%3A%5Bhigh%2C%20medium%5D%20not%20found%20after%20recompute&referrer=issue-stream&stream_index=1)
(23 events in 90d)

We have a structural issue with cached data and our locking mechanism:
if a data is missing in cache and queried at the same time by two
different entities, the first one will recompute the data and indicate a
lock during the operation, while the second one will seek to recompute
the data too but be stopped because of the ongoing lock, and be left
with no data to use, condemned to throw an error. In this PR I
considered that it was more important to avoid that error and chose to
ignoreLock instead when the data is being queried (through
getFromCacheWithRecompute), but this is maybe questionnable.
An important note is that I can't figure out how users that regularly
use twenty (as it has been the case since this error occured on our
internal workspace) can encounter that error as once computed, **the key
should always be present in the workspace**: the corresponding value it
is always updated, never deleted (until this PR) and recreated. I was
not able understand how this happened

For our data cached without a version to refer to in the database, I
also chose to ignore the lock when the recompute is triggered by a data
change (eg feature flag enabling or assigning user to a role or adding
an objectPermission on a role, etc.), as we never want to imped the
recompute in that case to avoid potential stale data.
2025-05-14 15:41:24 +02:00
4d78f5f97f [permissions] Improve performances using a cache for userWorkspaces roles (#11587)
In this PR we are 

- introducing a cached map `{ userworkspaceId: roleId } `to reduce calls
to get a userWorkspace's role (we were having N+1 around that with
combinedFindMany queries and generally having a lot of avoidable
queries)
- using the roles permissions cache (`{ roleId: { objectNameSingular:
{ canRead: bool, canUpdate: bool, ...} } `) in Permissions V1's
userHasObjectPermission, in order to 1) improve performances to avoid
calls to get roles 2) start using our permissions cache
2025-04-16 17:07:43 +02:00
d4deca45e8 Read feature flags from cache (#11556)
We are now storing a workspace's feature flag map in our redis cache. 
The cache is invalidated upon feature flag update through the lab
resolver.
2025-04-14 17:31:13 +02:00