Skip to main content

Sync service

The Electric sync service is an Elixir application that manages active-active replication between your Postgres database and your local apps.

Components and connections

The main way to run the sync service is using the official Docker image. See Usage -> Installation -> Sync service for more instructions.

Configuration options are passed as environment variables, e.g.:

docker run \
-e "DATABASE_URL=postgresql://..." \
-e "ELECTRIC_WRITE_TO_PG_MODE=direct_writes" \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_JWT_ALG=HS512" \
-e "AUTH_JWT_KEY=..." \
-p 5133:5133 \
-p 5433:5433 \
electricsql/electric

This page documents all the configuration options, including the:

info

For a longer form description of how to successfully deploy the sync service and what needs to connect where, see Deployment -> Concepts.

A note on ports

Your configuration options affect the number and type of ports that need to be exposed. You must always expose the HTTP_PORT. Your write-to-PG mode and migrations proxy config then determines whether you need to expose up-to two TCP ports: the LOGICAL_PUBLISHER_PORT and PG_PROXY_PORT respectively.

In development using Docker you usually want to map all the necessary ports to your host network (-p 5133:5133 and -p 5433:5433 in the example above).

In production you must make sure your hosting infrastructure exposes the necessary ports and protocols. If not, you can use the workarounds provided in the form of direct writes mode and the proxy tunnel.

Again, see Deployment -> Concepts for more information.

Core config

Configure how Electric connects to Postgres and exposes its HTTP/WebSocket API.

DATABASE_URL

VariableDATABASE_URL required
Description

Postgres connection string. Used to connect to the Postgres database.

The connection string must be in the libpg Connection URI format of postgresql://[userspec@][hostspec][/dbname][?sslmode=<sslmode>].

The userspec section of the connection string specifies the database user that Electric connects to Postgres as. The permissions required for this user depend on your choice of write-to-PG-mode. See the database user permissions section below for more information.

The optional sslmode query parameter may be set to one of the following values:

  • disable
  • allow
  • prefer
  • require

Including sslmode=require in the database connection string is equivalent to setting DATABASE_REQUIRE_SSL=true (which also happens to be the default). Any other value for sslmode is equivalent to setting DATABASE_REQUIRE_SSL=false. Do note that if you explicitly set DATABASE_REQUIRE_SSL, the sslmode query parameter in DATABASE_URL will be ignored.

ExampleDATABASE_URL=postgresql://user:password@example.com:54321/electric

DATABASE_REQUIRE_SSL

VariableDATABASE_REQUIRE_SSL
Defaulttrue
Description

Set to false to enable Electric to fallback to using unencrypted connections in case the database is not configured to work with SSL.

Be mindful of changing this default, more often than not it's a bad idea to use unencrypted database connections because all data flowing between your database and Electric may get intercepted by an unauthorized party.

Whether Electric will use SSL encryption for its database connections can also be configured with the sslmode query parameter in DATABASE_URL but only if DATABASE_REQUIRE_SSL is not explicitly set.

ExampleDATABASE_REQUIRE_SSL=false

DATABASE_USE_IPV6

VariableDATABASE_USE_IPV6
Defaultfalse
Description

Set to true to prioritise connecting to the database over IPv6. Electric will fall back to an IPv4 DNS lookup if the IPv6 lookup fails.

ExampleDATABASE_USE_IPV6=true

ELECTRIC_USE_IPv6

VariableELECTRIC_USE_IPV6
Defaulttrue
Description

Set to false to force Electric to only listen on IPv4 interfaces.

By default, Electric will accept inbound connections over both IPv6 and IPv4 when running on Linux. On Windows and some BSD systems inbound connections over IPv4 will not be accepted unless this option is disabled.

ExampleELECTRIC_USE_IPV6=false

HTTP_PORT

VariableHTTP_PORT
Default5133
Description

Port for HTTP connections.

This exposes an HTTP API used by the CLI and Generator on /api and a WebSocket API used by the Satellite replication protocol on /ws.

When using the Proxy tunnel, connections to the Migrations proxy are also tunneled over this port.

ExampleHTTP_PORT=8080

ELECTRIC_RESUMABLE_WAL_WINDOW

VariableELECTRIC_RESUMABLE_WAL_WINDOW
Default2G
Description

Maximum size that the sync service is allowed to use for keeping around old WAL records in Postgres. Measured in bytes unless one of the following suffixes is used: k for KB; K for KiB; m for MB; M for MiB; g for GB; G for GiB.

Normally, Postgres discards WAL records as soon as they are acknowledged by the replica. However, for the sync service to be able to restore its caches after a restart, it needs to hold on to old WAL records since those may contain transactions that affect electrified tables.

Setting this to a low value may lead to clients having to discard their local copy of the server state and restart their replication stream from scratch.

In the future Electric may store change diffs or employ techniques for compacting the retained WAL records with the goal of extending the time frame during which a client may catch up to the latest server state without discarding its local state.

ExampleELECTRIC_RESUMABLE_WAL_WINDOW=800M

ELECTRIC_TXN_CACHE_SIZE

VariableELECTRIC_TXN_CACHE_SIZE
Default256M
Description

Size of the in-memory cache for transactions the sync service consumes from Postgres over the logical replication stream. Measured in bytes unless one of the following suffixes is used: k for KB; K for KiB; m for MB; M for MiB; g for GB; G for GiB.

This cache is used to quickly catch up newly connected clients to the current state in Postgres. When the sync service restarts, it will repopulate this in-memory cache by streaming already committed transactions from retained Postgres' WAL records, going as far back in time as the size of the WAL window configured with ELECTRIC_RESUMABLE_WAL_WINDOW allows.

ExampleELECTRIC_TXN_CACHE_SIZE=2g

LOG_LEVEL

VariableLOG_LEVEL
Defaultinfo
Description

Verbosity of Electric's log output.

Available levels, in the order of increasing verbosity:

  • error
  • warning
  • info
  • debug
ExampleLOG_LEVEL=debug

Write-to-PG mode

Electric writes data to Postgres using one of two modes:

  1. logical replication
  2. direct writes
caution

The mode you choose affects your networking config and database user permissions.

ELECTRIC_WRITE_TO_PG_MODE

VariableELECTRIC_WRITE_TO_PG_MODE
Defaultlogical_replication
Description

The mode to use when writing data from Electric to Postgres. The allowed values are: logical_replication and direct_writes.

In logical_replication mode, Electric provides a logical replication publisher service over TCP that speaks the Logical Streaming Replication Protocol. Postgres connects to Electric and establishes a subscription to this. Writes are then streamed in and applied using logical replication.

In direct_writes mode, Electric writes data to Postgres using a standard interactive client connection. This avoids the need for Postgres to be able to connect to Electric and reduces the permissions required for the database user that Electric connects to Postgres as.

ExampleELECTRIC_WRITE_TO_PG_MODE=direct_writes

Logical replication mode

In logical_replication mode, Electric exposes a logical replication publisher service over TCP that speaks the Logical Streaming Replication Protocol.

Postgres connects to Electric on LOGICAL_PUBLISHER_HOST:LOGICAL_PUBLISHER_PORT and establishes a logical replication subscription to this publisher service. Writes are then streamed in and applied using the logical replication subscription.

         | <----------- DATABASE_URL ----------- |
Postgres | | Electric
| ---- LOGICAL_PUBLISHER_HOST:PORT ---> |
Caution

In logical replication mode:

  1. the database user that Electric connects to Postgres as must have the SUPERUSER role attribute
  2. Postgres must be able to connect to Electric (i.e.: must be able to establish an outbound TCP connection) on the host and port that you configure

As a result, you must make sure (in terms of networking / firewalls) not only that Postgres is reachable from Electric but also that Electric is reachable from Postgres. And Electric must know its own address, in order to provide it to Postgres when setting up the logical replication publication that allows writes to be replicated into Postgres.

LOGICAL_PUBLISHER_HOST

VariableLOGICAL_PUBLISHER_HOST required
Description

Hostname of the electric instance that Postgres connects when establishing the logical replication subscription.

Electric must be accessible from the Postgres instance on this hostname.

Required when-and-only-when running in logical replication mode.

ExampleLOGICAL_PUBLISHER_HOST=example.com

LOGICAL_PUBLISHER_PORT

VariableLOGICAL_PUBLISHER_PORT
Default5433
Description

Port number of the TCP service exposed by Electric that Postgres connects to when establishing the logical replication subscription.

The port must be accessible from the Postgres instance.

ExampleLOGICAL_PUBLISHER_PORT=65433

Direct writes mode

In direct_writes mode, Electric writes data to Postgres using a standard interactive client connection. This avoids the need for Postgres to be able to connect to Electric and reduces the permissions required for the database user that Electric connects to Postgres as.

Postgres | <----------- DATABASE_URL ----------- | Electric

No additional configuration is required for direct writes mode. The LOGICAL_PUBLISHER_HOST and LOGICAL_PUBLISHER_PORT variables are not required and are ignored.

Why are there two modes?

Originally (prior to v0.8) all writes to Postgres were made using logical replication.

In version 0.8, Electric added direct writes mode to reduce the database user permissions required and increase compatibility with Postgres hosting providers.

In future, we may deprecate logical replication and consolidate on direct writes. However, logical replication may have performance advantages and, for now, we've kept both modes available while direct writes mode gains maturity and we learn more about the operational characteristics of both modes.

Database user permissions

The Electric sync service connects to Postgres using the DATABASE_URL connection string, in the format postgresql://[userspec@][hostspec][/dbname].

The userspec section of this connection string specifies the database user that Electric connects to Postgres as. This user must have the following permissions.

Permissions for logical replication mode

In logical replication mode, the database user must have the LOGIN and SUPERUSER role attributes. You can create a user with these permissions using, e.g.:

CREATE ROLE electric
WITH LOGIN
PASSWORD '...'
SUPERUSER;

Permissions for direct writes mode

In direct writes mode, the database user must have LOGIN, REPLICATION and then either ALL on the database and public schema or at a minimum:

  • CONNECT, CREATE and TEMPORARY on the database
  • CREATE, EXECUTE on ALL and USAGE on the public schema

Plus ALTER DEFAULT PRIVILEGES to grant the same permissions on any new tables in the public schema. For example, to create a user with the necessary permissions:

CREATE ROLE electric
WITH LOGIN
PASSWORD '...'
REPLICATION;

GRANT ALL
ON DATABASE '...'
TO electric;

GRANT ALL
ON ALL TABLES
IN SCHEMA public
TO electric;

ALTER DEFAULT PRIVILEGES
IN SCHEMA public
GRANT ALL
ON TABLES
TO electric;

Migrations proxy

Electric exposes a Migrations proxy as a TCP service. This must be secured using PG_PROXY_PASSWORD and is exposed on PG_PROXY_PORT.

Connecting to the migrations proxy over a TCP port

The PG_PROXY_PORT supports a special http value that allows you to connect to the migrations proxy over a TCP-over-HTTP tunnel. This enables the use of the Proxy Tunnel. This is a CLI command that tunnels the Migrations proxy connection over the HTTP_PORT.

Connecting to the migrations proxy using a Proxy tunnel

PG_PROXY_PASSWORD

VariablePG_PROXY_PASSWORD required
Description

Password to use when connecting to the Postgres proxy via psql or any other Postgres client.

ExamplePG_PROXY_PASSWORD=b3aed739144e859a

PG_PROXY_PORT

VariablePG_PROXY_PORT
Default65432
Description

Port number for connections to the Migrations proxy.

Electric provides a migrations proxy service over TCP that speaks the Postgres protocol. This configures the port that this service is exposed on.

If you have Electric deployed behind a restrictive firewall that only allows HTTP/HTTPS connections, you can set the value to http or add http: as a prefix to the port number, e.g. http:1234. This will enable tunelling mode in which the migrations proxy will accept WebSocket connections from the Proxy tunnel while the TCP service will be listening for connections on port 1234.

Setting the value to http makes the TCP service listen on the default port.

ExamplePG_PROXY_PORT=http:65432

Authentication

Electric provides two authentication modes:

  1. secure (the default)
  2. insecure

In secure more, AUTH_JWT_ALG and AUTH_JWT_KEY are required. In insecure mode, all other authentication variables can be omitted.

AUTH_MODE

VariableAUTH_MODE
Defaultsecure
Description

Authentication mode to use to authenticate clients. The allowed values are: secure and insecure.

Secure mode is the default and is strongly recommended for production use. Insecure mode should only be used in development.

ExampleAUTH_MODE=insecure

AUTH_JWT_NAMESPACE

VariableAUTH_JWT_NAMESPACE optional
Description

This is an optional setting that specifies the location inside the token of custom claims that are specific to Electric.

Currently, only the sub custom claim (formerly user_id) is required.

ExampleAUTH_JWT_NAMESPACE=example

Secure mode

In secure mode, Electric authenticates its replication connections by obtaining a JWT from each client and verifying its validity before allowing data streaming in either direction.

See Usage -> Authentication -> Secure mode

AUTH_JWT_ALG

VariableAUTH_JWT_ALG required
Description

The algorithm to use for JWT verification. The following algorithms are supported:

  • HS256, HS384, HS512: HMAC-based cryptographic signature that relies on the SHA-2 family of hash functions
  • RS256, RS384, RS512: RSA-based algorithms for digital signature
  • ES256, ES384, ES512: ECC-based algorithms for digital signature
ExampleAUTH_JWT_ALG=HS512

AUTH_JWT_KEY

VariableAUTH_JWT_KEY required
Description

The key to use for JWT verification.

Must be appropriate for the chosen signature algorithm.

For HS* algorithms, the symmetric key can be base64-encoded, provided that you also configure AUTH_JWT_KEY_IS_BASE64_ENCODED.

For RS* and ES* algorithms, the public key must be in the PEM format:

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQY...
...
-----END PUBLIC KEY-----
ExampleAUTH_JWT_KEY=x\a0CA7Q4_gc...

AUTH_JWT_KEY_IS_BASE64_ENCODED

VariableAUTH_JWT_KEY_IS_BASE64_ENCODED optional
Description

A boolean value that indicates whether the key configured in AUTH_JWT_KEY needs to be base64-decoded before it is used for token signature verification.

ExampleAUTH_JWT_KEY_IS_BASE64_ENCODED=true

AUTH_JWT_ISS

VariableAUTH_JWT_ISS optional
Description

Specify the "issuer" that will be matched against the iss claim extracted from JWT auth tokens. This can be used to ensure that only tokens created by the expected party are used to authenticate your client.

ExampleAUTH_JWT_ISS=example.com

AUTH_JWT_AUD

VariableAUTH_JWT_AUD optional
Description

Specify the "audience" that will be matched against the aud claim extracted from JWT auth tokens. This can be used to ensure that only tokens for a specific application are used to authenticate your client.

ExampleAUTH_JWT_AUD=example.com

Insecure mode

Insecure mode is designed for development or testing. It supports unsigned JWTs that can be generated anywhere, including on the client, as well as signed JWTs which are accepted with no signature verification.

All other authentication variables (aside from AUTH_MODE) can be omitted.

See Usage -> Authentication -> Insecure mode for more information.

Telemetry

By default, ElectricSQL collects aggregated, anonymous usage data and sends them to our telemetry service. See Reference -> Telemetry for more information.

ELECTRIC_TELEMETRY

VariableELECTRIC_TELEMETRY
Defaultenabled
Description

Telemetry mode. The allowed values are: enabled and disabled.

Telemetry is enabled by default. Set to disabled to disable collection.

It's extremely helpful to leave telemetry enabled if you can.

ExampleELECTRIC_TELEMETRY=disabled