ClickHouse and custom reports: configuration via pyrus-setup

Custom reports are designed to analyze exactly the metrics that matter to your company. They let you see not only the current status of tasks, but also the entire history of actions — this gives a much more complete picture than standard exports.

Learn more about the purpose and benefits of custom reports

The reporting engine runs on top of ClickHouse. All configuration operations are performed using a dedicated migrator, pyrus-setup. It runs as a one-off container via the docker-compose run --rm command.

1. Environment variables for the ClickHouse container

  clickhouse:
    image: cr.yandex/crpl4n1nijtqddto2029/dev-pyrus-clickhouse:1.26.0
    container_name: pyrus-clickhouse
    environment:
      - CLICKHOUSE_USER=default
      - CLICKHOUSE_PASSWORD=default
      - CLICKHOUSE_DB=public
      - CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
    volumes:
      - pyrus_clickhouse_data:/var/lib/clickhouse
      - pyrus_clickhouse_logs:/var/log/clickhouse-server
    networks:
      - pyrus-backend

Creating users, granting permissions, and changing passwords are standard ClickHouse operations. Here are some useful links to the documentation:

• Users and passwords in users.xml: https://clickhouse.com/docs/operations/settings/settings-users
• CREATE USER: https://clickhouse.com/docs/sql-reference/statements/create/user
• ALTER USER (including for changing a password): https://clickhouse.com/docs/sql-reference/statements/alter/user
• SQL-based access management: https://clickhouse.com/docs/operations/access-rights

If you change the password, don't forget to pass the new credentials to Pyrus. To do this, use the enable-report-engine command from the next section.

2. enable-report-engine — enabling and configuring

What the command does:

1. saves the ClickHouse connection settings in the Pyrus configuration;
2. brings the ClickHouse schema up to the current version (applies migrations);
3. enables the ReportEngine feature flag;
4. starts the online crawler that transfers new events into the analytics database.

The command is safe to run repeatedly — you can run it several times without any negative consequences.

Parameters:

How values are determined: explicit argument → value already saved in the configuration → the standard default.

If a parameter is not passed, it is not overwritten.

SingleNode:

docker-compose run --rm pyrus-setup enable-report-engine \
  --clickhouseMode SingleNode \
  --clickhouseHost pyrus-clickhouse \
  --clickhousePort 8123 \
  --clickhouseUser default \
  --clickhousePassword 'PASSWORD'

Replicated (cluster):

docker-compose run --rm pyrus-setup enable-report-engine \
  --clickhouseMode Replicated \
  --clickhouseCluster CLUSTER_NAME \
  --clickhouseHost clickhouse-01 \
  --clickhousePort 8123 \
  --clickhouseUser pyrus_writer \
  --clickhousePassword 'PASSWORD'

If you only want to change the password in the Pyrus configuration — the remaining parameters will stay unchanged:

docker-compose run --rm pyrus-setup enable-report-engine \
  --clickhouseMode SingleNode \
  --clickhouseUser default \
  --clickhousePassword 'NEW_PASSWORD'

After changing the password, restart the consumers:

docker-compose restart pyrus-web-api pyrus-async-worker

3. disable-report-engine — disabling

What the command does: stops the online crawler and clears the ReportEngine feature flag.

The command has no parameters.

docker-compose run --rm pyrus-setup disable-report-engine

And if you just want to check the current state of the flag (without changing anything):

docker-compose run --rm pyrus-setup --setflag -f ReportEngine -v true -ls

4. --notes-to-analytic — indexing history

What the command does: transfers already accumulated historical task comments into the ClickHouse analytics database. When you enable the reporting engine, it processes only new events. To be able to build reports on historical data as well, this separate run is required.

The process can be interrupted and resumed — the position is set by the -start cursor.

Parameters:

Trial run for a single form only:

docker-compose run --rm pyrus-setup --notes-to-analytic -pc 2 -forms 12345

Full run:

docker-compose run --rm pyrus-setup --notes-to-analytic -pc 2

Run with increased load:

docker-compose run --rm pyrus-setup --notes-to-analytic -prof Medium -pc 4

Resuming after a stop:

docker-compose run --rm pyrus-setup --notes-to-analytic -pc 2 -start 372820809

5. Where the settings are stored in Pyrus

When you run enable-report-engine, the values are written into the Pyrus configuration. Here is which configuration key each parameter maps to:

Please note: the password is stored encrypted, so change it only through the enable-report-engine command. Do not try to edit it directly in the database or via SQL queries.