admin/twenty
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Fix cacheData not found after recomputes (#12032)

Browse Source 
Closes https://github.com/twentyhq/core-team-issues/issues/861

sentries: [User workspace role map not found after
recompute](https://twenty-v7.sentry.io/issues/6575092700/events/f9825338a30b470eb2345fe78c1e3479/?project=4507072499810304&query=is%3Aunresolved%20issue.priority%3A%5Bhigh%2C%20medium%5D%20not%20found%20after%20recompute&referrer=next-event&stream_index=0)
(64 events in 90d), [Feature flag map not found after
recompute](https://twenty-v7.sentry.io/issues/6547696076/?project=4507072499810304&query=is%3Aunresolved%20issue.priority%3A%5Bhigh%2C%20medium%5D%20not%20found%20after%20recompute&referrer=issue-stream&stream_index=1)
(23 events in 90d)

We have a structural issue with cached data and our locking mechanism:
if a data is missing in cache and queried at the same time by two
different entities, the first one will recompute the data and indicate a
lock during the operation, while the second one will seek to recompute
the data too but be stopped because of the ongoing lock, and be left
with no data to use, condemned to throw an error. In this PR I
considered that it was more important to avoid that error and chose to
ignoreLock instead when the data is being queried (through
getFromCacheWithRecompute), but this is maybe questionnable.
An important note is that I can't figure out how users that regularly
use twenty (as it has been the case since this error occured on our
internal workspace) can encounter that error as once computed, **the key
should always be present in the workspace**: the corresponding value it
is always updated, never deleted (until this PR) and recreated. I was
not able understand how this happened

For our data cached without a version to refer to in the database, I
also chose to ignore the lock when the recompute is triggered by a data
change (eg feature flag enabling or assigning user to a role or adding
an objectPermission on a role, etc.), as we never want to imped the
recompute in that case to avoid potential stale data.
This commit is contained in:
Marie
2025-05-14 15:41:24 +02:00
committed by GitHub
parent 83bdb1a515
commit 0c60fa9c23
10 changed files with 124 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
packages/twenty-server/src/engine/utils/get-data-from-cache-with-recompute.util.ts
View File

@ -1,3 +1,5 @@
import { Logger } from '@nestjs/common';
import { isDefined } from 'twenty-shared/utils';
import {
@ -17,36 +19,68 @@ const getFromCacheWithRecompute = async <T, U>({
recomputeCache,
cachedEntityName,
exceptionCode,
logger,
}: {
workspaceId: string;
getCacheData: (workspaceId: string) => Promise<U | undefined>;
getCacheVersion: (workspaceId: string) => Promise<T | undefined>;
recomputeCache: (params: { workspaceId: string }) => Promise<void>;
getCacheVersion?: (workspaceId: string) => Promise<T | undefined>;
recomputeCache: (params: {
workspaceId: string;
ignoreLock?: boolean;
}) => Promise<void>;
cachedEntityName: string;
exceptionCode: TwentyORMExceptionCode;
logger: Logger;
}): Promise<CacheResult<T, U>> => {
let cachedVersion: T | undefined;
let cachedData: U | undefined;
cachedVersion = await getCacheVersion(workspaceId);
const expectCacheVersion = isDefined(getCacheVersion);
if (expectCacheVersion) {
cachedVersion = await getCacheVersion(workspaceId);
}
cachedData = await getCacheData(workspaceId);
if (!isDefined(cachedData) || !isDefined(cachedVersion)) {
await recomputeCache({ workspaceId });
if (
!isDefined(cachedData) ||
(expectCacheVersion && !isDefined(cachedVersion))
) {
logger.warn(
`Triggering cache recompute for ${cachedEntityName} (workspace ${workspaceId})`,
{
cachedVersion,
cachedData,
},
);
await recomputeCache({ workspaceId, ignoreLock: true });
cachedData = await getCacheData(workspaceId);
cachedVersion = await getCacheVersion(workspaceId);
if (expectCacheVersion) {
cachedVersion = await getCacheVersion(workspaceId);
}
if (!isDefined(cachedData) || !isDefined(cachedVersion)) {
if (
!isDefined(cachedData) ||
(expectCacheVersion && !isDefined(cachedVersion))
) {
logger.warn(
`Data still missing after recompute for ${cachedEntityName} (workspace ${workspaceId})`,
{
cachedVersion,
cachedData,
},
);
throw new TwentyORMException(
`${cachedEntityName} not found after recompute for workspace ${workspaceId}`,
`${cachedEntityName} not found after recompute for workspace ${workspaceId} (missingData: ${!isDefined(cachedData)}, missingVersion: ${expectCacheVersion && !isDefined(cachedVersion)})`,
exceptionCode,
);
}
}
return {
version: cachedVersion,
version: cachedVersion as T,
data: cachedData,
};
};