In this PR:
- adding logs to track workspace creation performance
- refactor useIsWorkspaceSuspended to be more generic
- only fetch favorites and views if workspace is Active to avoid error
messages on sign up (workspace is not created yet)
Proposal:
- Add a method in ActiveWorkspaceCommand to loop over workspace safely
(add counter, add try / catch, provide datasource with fresh cache,
destroy datasource => as we do always do it)
Also in this PR:
- make sure we clear all dataSources (and not only the one on metadata
version in RAM)
In this huge (sorry!) PR:
- introducing objectMetadataItem in contextStore instead of
objectMetadataId which is more convenient
- splitting some big hooks into smaller parts to avoid re-renders
- removing Effects to avoid re-renders (especially onViewChange)
- making the view prefetch separate from favorites to avoid re-renders
- making the view prefetch load a state and add selectors on top of it
to avoir re-renders
As a result, the performance is WAY better (I suspect the favorite
implementation to trigger a lot of re-renders unfortunately).
However, we are still facing a random app freeze on view creation. I
could not investigate the root cause. As this seems to be already there
in the precedent release, we can move forward but this seems a urgent
follow up to me ==> EDIT: I've found the root cause after a few ours of
deep dive... an infinite loop in RecordTableNoRecordGroupBodyEffect...
prastoin edit: close https://github.com/twentyhq/twenty/issues/10253
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
Co-authored-by: prastoin <paul@twenty.com>
In this PR, I'm simplifying the lastVisitedView / Object logic:
- removing fallback logic as it's not useful
- splitting hooks into smaller hooks (to avoir re-renders)
- removing componentState on those states that are global
While making sure the upgrade-0.42 command was working as expected to
upgrade from 0.41 to 0.42, we've detected that the standardId of the new
bodyV2 (type RICH_TEXT_V2) was not properly set ; standardId was not
correct.
This was forcing the sync-metadata command to try to re-create the field
again.
We are using useInView to detect if a row should be rendered or not.
# Behavior issue
When browsing an Index page (let's say People), then navigating to
another one (Company), then back to People, the rows were not displayed
anymore. For some reason the inView value was set to false in this case
# Fix
- I have updated the useInView (react-intersection-observer) package but
it did not fix it
- useInView provides a ref. However, I believe this was conflicting with
the draggableRef; cecause we are in a <table> we cannot add additional
containers and are forced to apply both refs to the <tr> (draggableRef +
inViewRef). I believe this was causing the issue. I have added the
inView ref to an empty <td> within the row which is achieving the same
goal without forcing to combine refs
Fixes https://github.com/twentyhq/twenty/issues/9772
In this PR:
- the root cause of the issue that the ContextStoreViewIdEffect was not
filtering the views on objectMetadata properly
- I'm also deleting some over complex in the latestVisited view logic
- Duplicated logic between ContextStoreViewIdEffect and
ViewBarViewIdEffect, see my comment
We have recently introduced the possibility to sign up on Twenty cloud
without having to input a credit card which makes the demo.twenty.com
useless. Deprecating it!
In this PR:
- create a command to remove duplicate (groupBy messageId,
messageChannelId) within messageChannelmessageAssociaitions.
THis is needed to apply a constraint already present on
messageChannelmessageAssociaitions (to be executed before syncing
metadata)
I've re-investigated the captchaToken being invalid on workspace domain
url in case where email is passed in the URL.
We need to be a bit more granular
In this PR:
- removing rootDir / baseUrl from any tsconfig.json
- we need to keep it in tsconfig.spec.json and also specify rootDir in
jest.config.ts moduleMapper because of the way nx jest executor works
(automatically moving back to root)
- we need to explictly specify the depencies to twenty-shared /
twenty-emails (built packages) in packages package.json to help nx
understand dependencies
In this PR:
- migrate WorkspaceActivationStatus to twenty-shared (and update case to
make FE and BE consistent)
- introduce isWorkspaceActiveOrSuspended in twenty-shared
- refactor the code to use it (when we fetch data on the FE, we want to
keep SUSPENDED workspace working + when we sync workspaces we want it
too)
In this PR:
- remove old versions upgrade commands
- add a 0.40 upgrade command to loop over all INACTIVE workspaces and
either: update to SUSPENDED (if workspaceSchema exists), update them to
SUSPENDED + deletedAt (if workspaceSchema does not exist anymore)
Note: why updating the deleted one to SUSPENDED? Because I plan to
remove INACTIVE case in the enum in 0.41
Tests made on production like database:
- dry-mode
- singleWorkspaceId
- 3 cases : suspended, deleted+suspended, deleted+suspended+delete all
data
We are introducing a new workspace activationStatus "SUSPENDED". This
status represents a workspace which is SUSPENDED (either manually by the
admin or in case if IS_BILLING_ENABLED if the subscription is unpaid |
canceled | paused).
We will keep making sure these workspaces are healthy but prevent the
user from using it (they will be redirected to the billing page)
There are two follow ups to this PR:
- Bug: sometimes when opening Cmd+K from a deleted record, we are facing
a global error
- On Index page, actions in top right are displaying label and not short
name
- Implement multiple actions once refactoring on delete is complete
---------
Co-authored-by: bosiraphael <raphael.bosi@gmail.com>
We have recently introduced the possibility to specify workspace
specific auth providers.
I'm:
- introducing system wide auth providers (provided by clientConfig)
- making sure workspace specific auth providers belong to system wide
auth providers set
We were using RecordIndexContext in FieldDisplays components which is
wrong as the FieldDisplays could be used in locations not providing this
context (not being indexes pages).
Instead, we are passing it within FieldContext which the context that
will always be there for FieldDisplays and we need indexes to fill this
FieldContext with their RecordIndexContext value when needed
In this PR, I'm
- removing setting up the isAppWaitingForFreshMetadata boolean state in
PageChangeEffect navigate (not robust) to some precise synchronous
places, improving the control we have on when the app considers it's
ready to be rendered based on fresh metadata
- fixing tests
Fixes#8601
We had 3 implementations of getImageAbsoluteURI: in twenty-front, in
twenty-ui and in twenty-emails. I was able to remove the one in
twenty-front but I could not remove it from twenty-emails as this is a
standalone for now. The vision is to introduce shared utils in a
twenty-shared package
This PR is fixing the Task/Note Target picker.
## Issue
A few weeks ago, we have simplified the recordPicker logic. During this
refacto, we stopped making sure the selected records were fetched.
However, the optimistic update of the activity supposed that current
activityTargets are present in the selection in the picker in order to
filter out the removed items.
## Fix
In case we don't find the record in the picker selection, we don't
filter it out (it means the user cannot have removed it)
## Next step
@ijreilly I think we should put it back, as the selected records do not
appear on top anymore
## Issue
Some users were facing issues while updating SELECT or MULTISELECT
options:
```
Error executing migration QueryFailedError: column "undefined" does not exist
at PostgresQueryRunner.query (/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async WorkspaceMigrationEnumService.migrateEnumValues (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/services/workspace-migration-enum.service.js:101:13)
at async WorkspaceMigrationEnumService.alterEnum (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/services/workspace-migration-enum.service.js:54:9)
at async WorkspaceMigrationRunnerService.alterColumn (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:254:13)
at async WorkspaceMigrationRunnerService.handleColumnChanges (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:202:21)
at async WorkspaceMigrationRunnerService.handleTableChanges (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:94:25)
at async WorkspaceMigrationRunnerService.executeMigrationFromPendingMigrations (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:60:17)
at async WorkspaceExecutePendingMigrationsCommand.run (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/commands/workspace-execute-pending-migrations.command.js:24:9)
at async Command.<anonymous> (/app/node_modules/nest-commander/src/command-runner.service.js:162:24) {
query: '\n' +
' UPDATE "workspace_7i7bwawp7keh3gel1g69jropc"."_funnelInProductsRu"\n' +
' SET "reasonForStageTransition" = undefined\n' +
" WHERE id='2af8db61-5f75-46de-8b1a-97e312937e06'\n" +
' ',
parameters: undefined,
driverError: error: column "undefined" does not exist
```
## Root cause
Internally, while migrating enum values, we are doing:
```
const values = await queryRunner.query(
`SELECT id, "${oldColumnName}" FROM "${schemaName}"."${tableName}"`,
);
let val = value[oldColumnName];
```
oldColumnName being: `${columnDefinition.columnName}_old_${v4()}`;
However, TypeORM only supports 63 bits identifiers:
https://github.com/typeorm/typeorm/issues/3118, but silently truncate
the identifer so if oldColumnName gets larger than 63 characters,
`value[oldColumnName]` is undefined.
## Fix
We only need a temporary and unique temporary name, no need to take into
account the previous columnName here
I'm updating the docs as we now require the People API to be available
to use messaging sync. This has been reported by a user self-hosting the
app on discord.
On the kanban page, the record creation was changed a few weeks ago to
enable creation on top / bottom of the columns.
However, this introduced a glitch (missing background opacity). While
fixing it, I have refactored the component structure to:
- separate "New" button from the Empty record card
- Removing unique constraint creation in 0.32 as we have a dependency on
sync-metadata and sync-metadata has a dependency on it if we keep it.
- sync-metadata does not support well the deletion of activity +
activityTarget, silently swallow the exception for now
We have previously fixed the unselection of table records on click
outside. However, the ref was mispositioned as it selected the full
height table. In the case of low record numbers, we also want the
unselection to happen on table body click
Our postgres column naming convention is camelCase ; this forces SQL
queries to wrap column names with double quotes.
We previously forgot the quotes in a filter parsing case
In this PR, I'm fixing part of the impact of soft deletion on optimistic
rendering.
## Backend Vision
1) Backend endpoints will not return soft deleted records (having
deletedAt set) by default. To get the softDeleted records, we will pass
a { withSoftDelete: true } additional param in the query.
2) Record relations will NEVER contain softDeleted relations
## Backend current state
Right now, we have the following behavior:
- if the query filters do not mention deletedAt, we don't return
softDeletedRecords
- if the query filters mention deletedAt, we take it into consideration.
Meaning that if we want to have the softDeleted records in any way we
need to do { or: [ deletedAt: NULL, deletedAt: NOT_NULL] }
## Optimistic rendering strategy
1) useDestroyOne/Many is triggering destroyOptimisticEffects (previously
deleteOptimisticEffects)
2) UseDeleteOne/Many and useRestoreOne/Many are actually triggering
updateOptimisticEffects (as they only update deletedAt field) AND we
need updateOptimisticEffects to take into account deletedAt (future
withSoftDelete: true) filter.
While rebuilding the new GraphQLAPI (without pg_graphql), we forgot to
include the FieldGetter logic. This logic will soon be moved at ORM
level but we will need to keep it there for now
The bug has many impacts such as the fileToken not being generated and
preventing users from loading files
In this PR:
- remove deprecated EMAIL, PHONE, LINK field types (except for Zapier
package as there is another work ongoing)
- remove composite currency filter on currencyCode, actor filter on name
and workspaceMember as the UX is not great yet
## Context
We have recently merged a refactoring of our view module. However, one
case was forgotten which is to test our dynamic filtering logic.
It is currently possible to pass unsaved filters through the URL and
these filters will be applied to the currentView through
`QueryParamsFiltersEffect`. This component was saving filters but also
listening to them through useGetCurrentView hook.
## How
1) I'm removing this infinite loop by directly loading currentViewId
through the right recoil atom.
Bonus: I'm also removing the unmounting logic which seems wrong to me as
unsaved filters are mounted on a specific view, there is no need to
remove them while switching views in my opinion.
Fix all the broken CIs :p
This includes an ongoing effort to simplify test maintenance by having 1
unique source of truth about metadata and data mocks (that will later be
generated from a unique source of seeds: dev = demo = test)
Regressions:
- Unit line coverage: 60 > 55
- Storybook Pages branch coverage: 40 > 35
We will need to write tests to increase those coverage
- RelationFieldDisplay perf: 0.2ms to 0.22ms > We might have a
regression here
- Removed perf story about RawJSON > We will need to re-add it
We are updating our git worklow.
Case 1: **URGENT / PATCH**
If you want to include something URGENT that cannot wait for the next
release, you'll need to:
- create a PR from the latest patch (right now v0.30.1)
- create a new patch tag from this PR (would be v0.30.2 right now)
- merge this PR in main so it's in 0.31 too
Case 2: **REGULAR**
- Open a PR from main and merge it into main
I'm tagging main as v0.31.canary to make it clear!
This is a fast follow on v0.30 release:
- removing phone (deprecated PHONE field type) search from command menu.
I could have replaced it by a phone (PHONES field type) search but as we
are about to release the new search in v0.31 it does not seem to worse
the investment
- supporting EMAILS and PHONES field types in spreadsheet import
Note: while working on Spreadsheet import I found the code quite complex
and with areas having duplicated code. It does not seem to be a high
priority as I was able to maintain it at a low cost but it's not a
peaceful code surface to navigate!
In this PR:
- update your environment variables to default `CACHE_STORAGE_TYPE` to
`redis` and `MESSAGE_QUEUE_TYPE` to `bull-mq`
- add redis container to our default docker-compose
- add `REDIS_HOST` and `REDIS_PORT` to docker-compose yaml
- add upgrade instructions
We have a few issues on demo seeding:
- redis metdata cache was not flushed
- server ram graphql schema cache was not cleared on metadata version
increment
In this PR:
- removing ugprade-0.24 commands as we are releasing 0.30
- introducing cache:flush command
- refactoring upgrade command and sync-metadata command to use the
ActiveWorkspacesCommand so they consistently run on all workspaces or
selected workspaces
Fixes:
- clear localStorage on sign out
- fix missing workspaceMember in verify resolver
- do not throw on datasource already destroyed exception which can
happen with race condition when several resolvers are resolving in
parallel
In this PR:
1. Refactor guards to avoid duplicated queries: WorkspaceAuthGuard and
UserAuthGuard only check for existence of workspace and user in the
request without querying the database
This PR introduces the following changes:
- add the metadataVersion to all our metadata cache keys to ease
troubleshooting:
<img width="1146" alt="image"
src="https://github.com/user-attachments/assets/8427805b-e07f-465e-9e69-1403652c8b12">
- introduce a cache recompute lock to avoid overloading the database to
recompute the cache many time
We have recently merged: https://github.com/twentyhq/twenty/pull/6700
However, this introduced a regression on field edition as we have
removed the type dropdown from the field edition page. This dropdown was
wrapped into a controller setting the type on the form. Without this
type, the form is considered as invalid and cannot be saved.
I'm setting the form values through useForm.
I'm unhappy with this PR for too reasons:
- usage of activeMetadataField?.icon ?? '' format because I cannot call
useForm conditionnally. This would imply splitting the component into
several components to avoid this issue
- usage of react hook form which is very hard to debug, we should remove
it from the project
We had a regression on the record table as our inView hook was not able
to find the right div to compute its margin. This is because we are
identify this div by a hacky css selector as we don't have a direct
control on it (it's provided by our scrolling library)
Website CD has been broken by the recent addition of typeorm patch in
root package.json
Our current vision is to add npm package to each twenty-package
package.json directly
I have investigated the performance of our frontend vite build:
`npx nx run twenty:start` of `npx nx run twenty:build`
RAM usage:
- 160Mb: vite serve
- background typescript checker: 2.5GB
- background eslint checker: 3.5GB
I'm introducing two environment variables in FE .env to disable these
checkers on lower configuration (and to disable them from CD build):
```
# VITE_DISABLE_TYPESCRIPT_CHECKER=true
# VITE_DISABLE_ESLINT_CHECKER=true
```
WorkspaceMemberId is mandatory in the jwt token generated for a given
user on a given workspace.
However, when a user signs up, it does not have a workspaceMemberId yet.
Fix
[#web](https://github.com/orgs/twentyhq/projects/1/views/3?pane=issue&itemId=75329194)
This PR does 2 things:
- migrate webhooks to TwentyORM
- Fix inversion between objectNameSingular and operation in webhook
eventName. It is stored as {objectNameSingular}.{operation} and we were
querying {operation}.{objectNameSingular}
We have found the root cause of the issue:
- when using a datasource (including the cached ones), we are fetching
ObjectMetadataCollection from cache (700kB). Datasource usage is
happening any time we are using twentyORM, which is everywhere in the
jobs and in some resolvers (including the GetCurrentUser one). This is
leading to a high load on redis and leading to the performance issues we
are seeing.
- we actually don't need to fetch this objectMetadataCollection while
using a cached datasource, only when we instantiate a new one
## Context
As we grow, the messaging scripts are experiencing performance issues
forcing us to temporarily disable them on the cloud.
While investigating the performance, I have noticed that generating the
entity schema (for twentyORM) in the repository is taking ~500ms locally
on my Mac M2 so likely more on pods. Caching the entitySchema then!
I'm also clarifying naming around schemaVersion and cacheVersions ==>
both are renamed workspaceMetadataVersion and migrated to the workspace
table (the workspaceCacheVersion table is dropped).
This PR introduces the following changes:
- Add the ability to filter webhooks by objectSingularName and Actions
- Refactor SettingsWebhookDetails edition to not use react-hook-form
(which will be deprecated on the whole project)
- Updating the tests with a complex set of mock (we just need to fix ~30
of them now :D)
<img width="1053" alt="image"
src="https://github.com/user-attachments/assets/4e56d972-f129-4789-8d1c-4b5797a8ffd7">
In this PR:
- adding Favorites to Tasks and Notes
- fixing inconsistencies between custom object creation and sync of
standard fields of custom objects
- fixing workspaceCacheVersion not used to invalidate existing
datasource
In this PR:
- refactoring auth module to extract a jwt module that can be re-used
from other part of the app (avoiding circular dependencies file module
=> auth => file (file and auth both need jwt actually)
- activating imageIdentfier on person on workspace creation (this will
put back the images on people)
- fixing picture upload (we were missing some fileToken)
In this PR, I'm:
- adding createdBy field (type ACTOR) on custom objects when created
- moving `name` and `position` default column to the set of columns
automatically creation on object creation
- fixing a bug on mutations (update / create), if the targetted object
has a 'data' custom field, it was conflicting with the payload ==> I
feel we need to refactor this part of the code but we can keep this for
a bit later as we plan to move out of pg_graphql
<img width="1198" alt="image"
src="https://github.com/user-attachments/assets/891c4a97-bab1-415c-8551-dabd5996a794">
As per our guideline to maintain a smooth migration to the new minor
versions, this command is backfilling existing workspaces with the 3
userVars used to keep track of user onboarding:
```
ONBOARDING_CONNECT_ACCOUNT_COMPLETE = 'ONBOARDING_CONNECT_ACCOUNT_COMPLETE',
ONBOARDING_INVITE_TEAM_COMPLETE = 'ONBOARDING_INVITE_TEAM_COMPLETE',
ONBOARDING_CREATE_PROFILE_COMPLETE = 'ONBOARDING_CREATE_PROFILE_COMPLETE',
```
An ACTIVE workspace is a workspace that has a complete workspaceSchema
and is authorized to be browsed by users.
In this PR, I'm:
- introducing a new activationStatus: PENDING_CREATION (existing ACTIVE
/ INACTIVE)
- removing workspaceService.isWorkspaceActivated (based on
workspaceSchema existence which is not robust and checking
activationStatus.ACTIVE instead)
- removing dynamic activationStatus field on worksapce resolver (we can
use the postgres column directly now that data has been migrated)
- on user sign up creating the workspace in PENDING_CREATION, and on
workspace activation setting it to ACTIVE
- only re-activating a workspace if the current activationStatus is
INACTIVE through billing webhooks (a PENDING_CREATION should stay
PENDING and ACTIVE should stay ACTIVE)
In this PR, I'm fixing two issues that we have faced:
- computing a rich text first line in case of the first block content is
not a text
- migrating existing timelineActivities tied to tasks / notes to
linked-tasks / linked-notes during migration command
In this PR, I am:
- removing a useEffect (we use the onCompleted prop of useFindManyQuery
hook)
- moving this logic into a RecordShowPageHeader to avoid re-renders of
the whole tree
Charles Bochet