## Issue
Some users were facing issues while updating SELECT or MULTISELECT
options:
```
Error executing migration QueryFailedError: column "undefined" does not exist
at PostgresQueryRunner.query (/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async WorkspaceMigrationEnumService.migrateEnumValues (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/services/workspace-migration-enum.service.js:101:13)
at async WorkspaceMigrationEnumService.alterEnum (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/services/workspace-migration-enum.service.js:54:9)
at async WorkspaceMigrationRunnerService.alterColumn (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:254:13)
at async WorkspaceMigrationRunnerService.handleColumnChanges (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:202:21)
at async WorkspaceMigrationRunnerService.handleTableChanges (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:94:25)
at async WorkspaceMigrationRunnerService.executeMigrationFromPendingMigrations (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/workspace-migration-runner.service.js:60:17)
at async WorkspaceExecutePendingMigrationsCommand.run (/app/packages/twenty-server/dist/src/engine/workspace-manager/workspace-migration-runner/commands/workspace-execute-pending-migrations.command.js:24:9)
at async Command.<anonymous> (/app/node_modules/nest-commander/src/command-runner.service.js:162:24) {
query: '\n' +
' UPDATE "workspace_7i7bwawp7keh3gel1g69jropc"."_funnelInProductsRu"\n' +
' SET "reasonForStageTransition" = undefined\n' +
" WHERE id='2af8db61-5f75-46de-8b1a-97e312937e06'\n" +
' ',
parameters: undefined,
driverError: error: column "undefined" does not exist
```
## Root cause
Internally, while migrating enum values, we are doing:
```
const values = await queryRunner.query(
`SELECT id, "${oldColumnName}" FROM "${schemaName}"."${tableName}"`,
);
let val = value[oldColumnName];
```
oldColumnName being: `${columnDefinition.columnName}_old_${v4()}`;
However, TypeORM only supports 63 bits identifiers:
https://github.com/typeorm/typeorm/issues/3118, but silently truncate
the identifer so if oldColumnName gets larger than 63 characters,
`value[oldColumnName]` is undefined.
## Fix
We only need a temporary and unique temporary name, no need to take into
account the previous columnName here
I'm updating the docs as we now require the People API to be available
to use messaging sync. This has been reported by a user self-hosting the
app on discord.
On the kanban page, the record creation was changed a few weeks ago to
enable creation on top / bottom of the columns.
However, this introduced a glitch (missing background opacity). While
fixing it, I have refactored the component structure to:
- separate "New" button from the Empty record card
- Removing unique constraint creation in 0.32 as we have a dependency on
sync-metadata and sync-metadata has a dependency on it if we keep it.
- sync-metadata does not support well the deletion of activity +
activityTarget, silently swallow the exception for now
We have previously fixed the unselection of table records on click
outside. However, the ref was mispositioned as it selected the full
height table. In the case of low record numbers, we also want the
unselection to happen on table body click
Our postgres column naming convention is camelCase ; this forces SQL
queries to wrap column names with double quotes.
We previously forgot the quotes in a filter parsing case
In this PR, I'm fixing part of the impact of soft deletion on optimistic
rendering.
## Backend Vision
1) Backend endpoints will not return soft deleted records (having
deletedAt set) by default. To get the softDeleted records, we will pass
a { withSoftDelete: true } additional param in the query.
2) Record relations will NEVER contain softDeleted relations
## Backend current state
Right now, we have the following behavior:
- if the query filters do not mention deletedAt, we don't return
softDeletedRecords
- if the query filters mention deletedAt, we take it into consideration.
Meaning that if we want to have the softDeleted records in any way we
need to do { or: [ deletedAt: NULL, deletedAt: NOT_NULL] }
## Optimistic rendering strategy
1) useDestroyOne/Many is triggering destroyOptimisticEffects (previously
deleteOptimisticEffects)
2) UseDeleteOne/Many and useRestoreOne/Many are actually triggering
updateOptimisticEffects (as they only update deletedAt field) AND we
need updateOptimisticEffects to take into account deletedAt (future
withSoftDelete: true) filter.
While rebuilding the new GraphQLAPI (without pg_graphql), we forgot to
include the FieldGetter logic. This logic will soon be moved at ORM
level but we will need to keep it there for now
The bug has many impacts such as the fileToken not being generated and
preventing users from loading files
In this PR:
- remove deprecated EMAIL, PHONE, LINK field types (except for Zapier
package as there is another work ongoing)
- remove composite currency filter on currencyCode, actor filter on name
and workspaceMember as the UX is not great yet
## Context
We have recently merged a refactoring of our view module. However, one
case was forgotten which is to test our dynamic filtering logic.
It is currently possible to pass unsaved filters through the URL and
these filters will be applied to the currentView through
`QueryParamsFiltersEffect`. This component was saving filters but also
listening to them through useGetCurrentView hook.
## How
1) I'm removing this infinite loop by directly loading currentViewId
through the right recoil atom.
Bonus: I'm also removing the unmounting logic which seems wrong to me as
unsaved filters are mounted on a specific view, there is no need to
remove them while switching views in my opinion.
Fix all the broken CIs :p
This includes an ongoing effort to simplify test maintenance by having 1
unique source of truth about metadata and data mocks (that will later be
generated from a unique source of seeds: dev = demo = test)
Regressions:
- Unit line coverage: 60 > 55
- Storybook Pages branch coverage: 40 > 35
We will need to write tests to increase those coverage
- RelationFieldDisplay perf: 0.2ms to 0.22ms > We might have a
regression here
- Removed perf story about RawJSON > We will need to re-add it
We are updating our git worklow.
Case 1: **URGENT / PATCH**
If you want to include something URGENT that cannot wait for the next
release, you'll need to:
- create a PR from the latest patch (right now v0.30.1)
- create a new patch tag from this PR (would be v0.30.2 right now)
- merge this PR in main so it's in 0.31 too
Case 2: **REGULAR**
- Open a PR from main and merge it into main
I'm tagging main as v0.31.canary to make it clear!
This is a fast follow on v0.30 release:
- removing phone (deprecated PHONE field type) search from command menu.
I could have replaced it by a phone (PHONES field type) search but as we
are about to release the new search in v0.31 it does not seem to worse
the investment
- supporting EMAILS and PHONES field types in spreadsheet import
Note: while working on Spreadsheet import I found the code quite complex
and with areas having duplicated code. It does not seem to be a high
priority as I was able to maintain it at a low cost but it's not a
peaceful code surface to navigate!
In this PR:
- update your environment variables to default `CACHE_STORAGE_TYPE` to
`redis` and `MESSAGE_QUEUE_TYPE` to `bull-mq`
- add redis container to our default docker-compose
- add `REDIS_HOST` and `REDIS_PORT` to docker-compose yaml
- add upgrade instructions
We have a few issues on demo seeding:
- redis metdata cache was not flushed
- server ram graphql schema cache was not cleared on metadata version
increment
In this PR:
- removing ugprade-0.24 commands as we are releasing 0.30
- introducing cache:flush command
- refactoring upgrade command and sync-metadata command to use the
ActiveWorkspacesCommand so they consistently run on all workspaces or
selected workspaces
Fixes:
- clear localStorage on sign out
- fix missing workspaceMember in verify resolver
- do not throw on datasource already destroyed exception which can
happen with race condition when several resolvers are resolving in
parallel
In this PR:
1. Refactor guards to avoid duplicated queries: WorkspaceAuthGuard and
UserAuthGuard only check for existence of workspace and user in the
request without querying the database
This PR introduces the following changes:
- add the metadataVersion to all our metadata cache keys to ease
troubleshooting:
<img width="1146" alt="image"
src="https://github.com/user-attachments/assets/8427805b-e07f-465e-9e69-1403652c8b12">
- introduce a cache recompute lock to avoid overloading the database to
recompute the cache many time
We have recently merged: https://github.com/twentyhq/twenty/pull/6700
However, this introduced a regression on field edition as we have
removed the type dropdown from the field edition page. This dropdown was
wrapped into a controller setting the type on the form. Without this
type, the form is considered as invalid and cannot be saved.
I'm setting the form values through useForm.
I'm unhappy with this PR for too reasons:
- usage of activeMetadataField?.icon ?? '' format because I cannot call
useForm conditionnally. This would imply splitting the component into
several components to avoid this issue
- usage of react hook form which is very hard to debug, we should remove
it from the project
We had a regression on the record table as our inView hook was not able
to find the right div to compute its margin. This is because we are
identify this div by a hacky css selector as we don't have a direct
control on it (it's provided by our scrolling library)
Website CD has been broken by the recent addition of typeorm patch in
root package.json
Our current vision is to add npm package to each twenty-package
package.json directly
I have investigated the performance of our frontend vite build:
`npx nx run twenty:start` of `npx nx run twenty:build`
RAM usage:
- 160Mb: vite serve
- background typescript checker: 2.5GB
- background eslint checker: 3.5GB
I'm introducing two environment variables in FE .env to disable these
checkers on lower configuration (and to disable them from CD build):
```
# VITE_DISABLE_TYPESCRIPT_CHECKER=true
# VITE_DISABLE_ESLINT_CHECKER=true
```
WorkspaceMemberId is mandatory in the jwt token generated for a given
user on a given workspace.
However, when a user signs up, it does not have a workspaceMemberId yet.
Fix
[#web](https://github.com/orgs/twentyhq/projects/1/views/3?pane=issue&itemId=75329194)
This PR does 2 things:
- migrate webhooks to TwentyORM
- Fix inversion between objectNameSingular and operation in webhook
eventName. It is stored as {objectNameSingular}.{operation} and we were
querying {operation}.{objectNameSingular}
We have found the root cause of the issue:
- when using a datasource (including the cached ones), we are fetching
ObjectMetadataCollection from cache (700kB). Datasource usage is
happening any time we are using twentyORM, which is everywhere in the
jobs and in some resolvers (including the GetCurrentUser one). This is
leading to a high load on redis and leading to the performance issues we
are seeing.
- we actually don't need to fetch this objectMetadataCollection while
using a cached datasource, only when we instantiate a new one
## Context
As we grow, the messaging scripts are experiencing performance issues
forcing us to temporarily disable them on the cloud.
While investigating the performance, I have noticed that generating the
entity schema (for twentyORM) in the repository is taking ~500ms locally
on my Mac M2 so likely more on pods. Caching the entitySchema then!
I'm also clarifying naming around schemaVersion and cacheVersions ==>
both are renamed workspaceMetadataVersion and migrated to the workspace
table (the workspaceCacheVersion table is dropped).
This PR introduces the following changes:
- Add the ability to filter webhooks by objectSingularName and Actions
- Refactor SettingsWebhookDetails edition to not use react-hook-form
(which will be deprecated on the whole project)
- Updating the tests with a complex set of mock (we just need to fix ~30
of them now :D)
<img width="1053" alt="image"
src="https://github.com/user-attachments/assets/4e56d972-f129-4789-8d1c-4b5797a8ffd7">
In this PR:
- adding Favorites to Tasks and Notes
- fixing inconsistencies between custom object creation and sync of
standard fields of custom objects
- fixing workspaceCacheVersion not used to invalidate existing
datasource
In this PR:
- refactoring auth module to extract a jwt module that can be re-used
from other part of the app (avoiding circular dependencies file module
=> auth => file (file and auth both need jwt actually)
- activating imageIdentfier on person on workspace creation (this will
put back the images on people)
- fixing picture upload (we were missing some fileToken)
In this PR, I'm:
- adding createdBy field (type ACTOR) on custom objects when created
- moving `name` and `position` default column to the set of columns
automatically creation on object creation
- fixing a bug on mutations (update / create), if the targetted object
has a 'data' custom field, it was conflicting with the payload ==> I
feel we need to refactor this part of the code but we can keep this for
a bit later as we plan to move out of pg_graphql
<img width="1198" alt="image"
src="https://github.com/user-attachments/assets/891c4a97-bab1-415c-8551-dabd5996a794">
As per our guideline to maintain a smooth migration to the new minor
versions, this command is backfilling existing workspaces with the 3
userVars used to keep track of user onboarding:
```
ONBOARDING_CONNECT_ACCOUNT_COMPLETE = 'ONBOARDING_CONNECT_ACCOUNT_COMPLETE',
ONBOARDING_INVITE_TEAM_COMPLETE = 'ONBOARDING_INVITE_TEAM_COMPLETE',
ONBOARDING_CREATE_PROFILE_COMPLETE = 'ONBOARDING_CREATE_PROFILE_COMPLETE',
```
An ACTIVE workspace is a workspace that has a complete workspaceSchema
and is authorized to be browsed by users.
In this PR, I'm:
- introducing a new activationStatus: PENDING_CREATION (existing ACTIVE
/ INACTIVE)
- removing workspaceService.isWorkspaceActivated (based on
workspaceSchema existence which is not robust and checking
activationStatus.ACTIVE instead)
- removing dynamic activationStatus field on worksapce resolver (we can
use the postgres column directly now that data has been migrated)
- on user sign up creating the workspace in PENDING_CREATION, and on
workspace activation setting it to ACTIVE
- only re-activating a workspace if the current activationStatus is
INACTIVE through billing webhooks (a PENDING_CREATION should stay
PENDING and ACTIVE should stay ACTIVE)
In this PR, I'm fixing two issues that we have faced:
- computing a rich text first line in case of the first block content is
not a text
- migrating existing timelineActivities tied to tasks / notes to
linked-tasks / linked-notes during migration command
In this PR, I am:
- removing a useEffect (we use the onCompleted prop of useFindManyQuery
hook)
- moving this logic into a RecordShowPageHeader to avoid re-renders of
the whole tree
This PR refactors the ORM-Manager to simplify and unify the datasource
creation. I'm deprecating all usages if InjectWorkspaceDatasource and
InjectWorkspaceRepository as we can't be sure they are up-to-date
## Bug Description
We are facing a bug in case recaptcha is enabled.
To reproduce:
- Create your recaptcha: https://www.google.com/recaptcha/about/
- update your server .env with the following variables:
```
CAPTCHA_SECRET_KEY=REPLACE_ME
CAPTCHA_SITE_KEY=REPLACE_ME
CAPTCHA_DRIVER=google-recaptcha
```
- Go to the login page, enter an existing user email and hit 'Reset your
password'.
- Add a console.log in emailPasswordResetLink in auth.resolver.ts to get
the token that would be sent by email if you don't have the mailer setup
- Browse: /reset-password/{passwordToken}
- Update the password:
<img width="1446" alt="image"
src="https://github.com/user-attachments/assets/dd5b077f-293e-451a-8630-22d24ac66c42">
- See that the token is invalid
You should see two calls in your developer network tab. A successful one
to update the password and another to log you in. This 2nd call
(Challenge) does not have the captcha token provided. It should be
## Fix
- Refreshing the token on page load
- providing it to the Challenge graphql call
This PR fixes a few bugs on TwentyORM:
- fix many to one relations that were not properly queries
- fix many to one relations that were not properly parsed
- compute datasource (or use from cache) at run-time and do not use
injected one that could be outdated
We still need to refactor it to simplify, I feel the API are too complex
and we have too many cache layers. Also the relation computation part is
very complex and bug prone
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/9fcdd5ca-4329-467c-ada8-4dd5d45be259">
Open questions:
- the Tag component does not match Figma in term of style and API for
"transparent" | "outline". We need to discuss with @Bonapara what is the
desired behavior here
- right now opportunity.stage is not nullable. We need to discuss with
@FelixMalfait and @Bonapara what we want here. I would advocate to make
a it nullable for now until we introduce settings on select fields.
custom select are nullable and it could be confusing for the user
Follow up:
- enhance tests on Tags
- add story to cover the No Value column on record board
Using a portal in dropdown systematically can be an issue in case we are
having dropdown within dropdown. The useClickOutside listener will be
triggered. It's easier to usePortal only in the case we really need
them, which is quite rare
Recently, we've forced all dropdown menu to be displayed in portal. This
loses the z-index hieararchy structure and forces us to specify a higher
z-index in order for dropdown menus to be displayed on top
In this PR:
- Renaming SettingsAccountsEmailBlocklist to
SettingsAccountsEmailBlocklist as the blocklist is not tied to
emails/messaging but is user level
- Changing the UI settings UI by removing /emails/{id} page and adding
tabs on /emails page
<img width="1512" alt="image"
src="https://github.com/twentyhq/twenty/assets/12035771/d215a891-fff9-477d-915d-0d7a697742e8">
We have recently deprecated our subscriptionStatus on workspace to
replace it by a check on existing subscription (+ freeAccess
featureFlag) but the logic was not properly implemented
In this PR, I'm simplifying storybook setup:
1) Remove build --test configuration that prevent autodocs. We are not
using autodocs at all (the dev experience is not good enough), so I have
completely disabled it.
2) Clarify `serve` vs `test` vs `serve-and-test` configurations
After this PR:
- you can serve storybook in two modes: `npx nx run
twenty-front:storybook:serve:dev` and `npx nx run
twenty-front:storybook:serve:static`
- you can run tests agains an already served storybook (this is useful
in dev so you don't have to rebuild everytime to run tests): `npx nx run
twenty-front:storybook:test`
- you can conbine both: `npx nx run
twenty-front:storybook:serve-and-test:static`
As per title!
Also, I'm removing an incorrect logic in the enum migration runner that
takes care of the case where we have no defaultValue but non nullable
which is not a valid business case.
We had an issue on infinite scroll on table view.
The fetch more logic was modifying isTableLastRowVisible state (which is
wrong, how could it know)? This was done to prevent loading too much
data at once. This was causing some race condition on
isTableLastRowVisible (as the table itself was also changing it
depending on the real visibility of the line)
I have remove this hacky usage of isTableLastRowVisible and replaced it
by a setTimeout to let the user some time to scroll and introduce a
throttle logic.
Our tests on FE are red, which is a threat to code quality. I'm adding a
few unit tests to improve the coverage and lowering a bit the lines
coverage threshold
In this PR, I'm doing 2 things:
- refresh connectedAccount token on message-list-fetch. It's currently
only refresh while doing the messages-import. However messages-import
stage are only triggered if new messages are detected (which could take
days or week depending of the messageChannel activity). We should also
refresh it while trying to fetch the list
- handle Unhandled Gmail error code 500 with reason "backendError".
These can occur on gmail side. In this case, we just retry later.
In this PR, I'm mainly doing two things:
- uniformizing messaging-messages-import and
messaging-message-list-fetch behaviors (cron.job and job)
- improving performances of these cron.jobs by not triggering the jobs
if the stage is not relevant
- making sure these jobs have same signature (workspaceId +
messageChannelId)
In this PR, I'm refactoring the messaging module into smaller pieces
that have **ONE** responsibility: import messages, clean messages,
handle message participant creation, instead of having ~30 modules (1
per service, jobs, cron, ...). This is mandatory to start introducing
drivers (gmails, office365, ...) IMO. It is too difficult to enforce
common interfaces as we have too many interfaces (30 modules...). All
modules should not be exposed
Right now, we have services that are almost functions:
do-that-and-this.service.ts / do-that-and-this.module.ts
I believe we should have something more organized at a high level and it
does not matter that much if we have a bit of code duplicates.
Note that the proposal is not fully implemented in the current PR that
has only focused on messaging folder (biggest part)
Here is the high level proposal:
- connected-account: token-refresher
- blocklist
- messaging: message-importer, message-cleaner, message-participants,
... (right now I'm keeping a big messaging-common but this will
disappear see below)
- calendar: calendar-importer, calendar-cleaner, ...
Consequences:
1) It's OK to re-implement several times some things. Example:
- error handling in connected-account, messaging, and calendar instead
of trying to unify. They are actually different error handling. The only
things that might be in common is the GmailError => CommonError parsing
and I'm not even sure it makes a lot of sense as these 3 apis might have
different format actually
- auto-creation. Calendar and Messaging could actually have different
rules
2) **We should not have circular dependencies:**
- I believe this was the reason why we had so many modules, to be able
to cherry pick the one we wanted to avoid circular deps. This is not the
right approach IMO, we need architect the whole messaging by defining
high level blocks that won't have circular dependencies by design. If we
encounter one, we should rethink and break the block in a way that makes
sense.
- ex: connected-account.resolver is not in the same module as
token-refresher. ==> connected-account.resolver => message-importer (as
we trigger full sync job when we connect an account) => token-refresher
(as we refresh token on message import).
connected-account.resolver and token-refresher both in connected-account
folder but should be in different modules. Otherwise it's a circular
dependency. It does not mean that we should create 1 module per service
as it was done before
In a nutshell: The code needs to be thought in term of reponsibilities
and in a way that enforce high level interfaces (and avoid circular
dependencies)
Bonus: As you can see, this code is also removing a lot of code because
of the removal of many .module.ts (also because I'm removing the sync
scripts v2 feature flag end removing old code)
Bonus: I have prefixed services name with Messaging to improve dev xp.
GmailErrorHandler could be different between MessagingGmailErrorHandler
and CalendarGmailErrorHandler for instance
Charles Bochet