Error when trying to enable encryption in transit & start the Yugabyte on the 1st node

Hi, Im getting similar error. Im trying to enable encryption in transit & start the Yugabyte on the 1st node. Below are the details:

Operating System: Red Hat Enterprise Linux 8.10 (Ootpa)

– Command to start the yugabyte:

$HOME/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted start \
--config $HOME/yugabyte_start.conf \
--advertise_address=$(hostname) \
--cloud_location=aws.us-west-2.us-west-2a \
--base_dir=$HOME/yb_open_binaries \
--fault_tolerance=region
–Conf file

–Conf file

{
    "data_dir": "/crdb/data/yugabyte",
    "additional_data_dir": "",
    "log_dir": "/crdb/log/yugabyte",
    "gen_certs_dir": "/home/crdbadmin/yb_open_binaries/generated_certs",
    "master_rpc_port": 9092,
    "tserver_rpc_port": 9093,
    "master_webserver_port": 9094,
    "tserver_webserver_port": 9095,
    "ysql_port": 9090,
    "ycql_port": 9091,
    "ysql_metric_port": 9096,
    "ycql_metric_port": 12000,
    "advertise_address": "host_name",
    "webserver_port": 7200,
    "yugabyted_ui_port": 9097,
    "universe_uuid": "xxxxxx",
    "node_uuid": "xxxxxx",
    "tserver_uuid": "xxxxxx",
    "master_uuid": "xxxxxx",
    "placement_uuid": "xxxxxx",
    "polling_interval": "5",
    "callhome": true,
    "master_flags": "",
    "tserver_flags": "",
    "join": "",
    "ysql_enable_auth": true,
    "use_cassandra_authentication": true,
    "cloud_provider": "aws",
    "cloud_region": "us-west-2",
    "cloud_zone": "us-west-2a",
    "fault_tolerance": "region",
    "secure": true,
    "insecure": false,
    "certs_dir": "/home/crdbadmin/ybdb1d/certs",
    "ca_cert_file_path": "/home/crdbadmin/ybdb1d/certs/ca.crt",
    "database_password": null,
    "current_masters": "",
    "ui": true,
    "backup_daemon": true,
    "dns_enabled": true,
    "read_replica": false,
    "cluster_member": false

–Error:

Starting yugabyted…
:white_check_mark: YugabyteDB Started
/ Enabling Encryption in Transit and Password Authentication…Could not update Postgress user password. Exception: Traceback (most recent call last):
File “/home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted”, line 5617, in update_db_passwords
retry_op_with_argument(ysql_proxy.try_update_password, new_password, timeout=60)
File “/home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted”, line 10489, in retry_op_with_argument
now - start_time))
RuntimeError: Failed after retrying operation for 60.388899087905884 secs.

Im not able to upload the collect logs, pls guide a how to upload it.

Hi,

You can use yugabyted native support for enabling encryption in transit and authentication using --secure flag.

$HOME/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted start \
--config $HOME/yugabyte_start.conf \
--advertise_address=$(hostname) \
--cloud_location=aws.us-west-2.us-west-2a \
--base_dir=$HOME/yb_open_binaries \
--fault_tolerance=region
--secure
--config file

Are you trying to use custom certs and ca?

Thanks,
Nikhil

Yes, im trying to create a cluster with custom certs.

I have specified below in the conf file, please cross check if something needs to be changed:

"ysql_enable_auth": false,
"certs_dir": "/home/crdbadmin/ybdb1d/certs",
"ca_cert_file_path": "/home/crdbadmin/ybdb1d/certs",
"secure": true,
"insecure": false,

Hi Vineet2k1,

I believe this should have worked as you’ve specified the custom certs_dir where the certs are located. @vineet2k1 can you please send us the logs in YugabyteDB slack channel. For now, can you pls copy and paste the contents from base_dir/logs/yugabyted.log file to this thread.

@Sanskar_Garg if you have any additional info to share on this.

Thanks,
Nikhil Chandrappa

Hi Vineet2k1,

You can also upload the logs to google drive or any other online cloud storage provider and send us the link to download the logs.

Thanks,
Nikhil

Hi @nmalladi here is the yugabyted.log - yugabyted.log - Google Drive

Hi @vineet2k,

I went over the logs, YugabyteDB processes started successfully however yugabyted is not able to connect to YSQL to update the password which is failing after retrying for 60 secs. The Connection is getting refused for aws-us-west-2a-ybdb1d-crdb-01.g.xxxx.com:9090, I think this might be behind a firewall. do we know if the port is open for connections?

[yugabyted start] 2025-02-07 05:14:35,269 INFO:  | 12.0s | run_process: cmd: ['/home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin/ysqlsh', 'postgresql://yugabyte:[email protected]:9090', '-c', "alter role yugabyte password '************';"]
[yugabyted start] 2025-02-07 05:14:35,274 INFO:  | 12.0s | run_process returned 2: 
OUT >>

<< ERR >>
ysqlsh: error: connection to server at "aws-us-west-2a-ybdb1d-crdb-01.g.xxxx.com" (100.76.212.207), port 9090 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?

can you try an insecure cluster and see if you can connect to YSQL?

Yes @nmalladi the connectivity is fine, below are the details from the insecure cluster:

NON-PROD:!:_aws-us-west-2a-ybdb1d-crdb-01:/home/crdbadmin> $HOME/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted start
29:aws-us-west-2a-ybdb1d-crdb-01> --config $HOME/yugabyte_start.conf
29:aws-us-west-2a-ybdb1d-crdb-01> --advertise_address=$(hostname)
29:aws-us-west-2a-ybdb1d-crdb-01> --cloud_location=$(get_aws_region_az)
29:aws-us-west-2a-ybdb1d-crdb-01> --base_dir=$HOME/yb_open_binaries
29:aws-us-west-2a-ybdb1d-crdb-01> --fault_tolerance=region
Starting yugabyted…
:white_check_mark: YugabyteDB Started
:white_check_mark: UI ready
:white_check_mark: Data placement constraint successfully verified

:warning: WARNINGS:

  • open files ulimits value set low. Please set soft and hard limits to 1048576.
  • Cluster started in an insecure mode without authentication and encryption enabled. For non-production use only, not to be used without firewalls blocking the internet traffic.

Please review the following docs and rerun the start command:

±---------------------------------------------------------------------------------------------------------------------------------------+
| yugabyted |
±---------------------------------------------------------------------------------------------------------------------------------------+
| Status : Running. |
| YSQL Status : Ready |
| Replication Factor : 1 |
| YugabyteDB UI : http://aws-us-west-2a-ybdb1d-crdb-01.g.apple.com:9097 |
| JDBC : jdbc:postgresql://aws-us-west-2a-ybdb1d-crdb-01.g.apple.com:9090/yugabyte?user=yugabyte&password=yugabyte |
| YSQL : bin/ysqlsh -h aws-us-west-2a-ybdb1d-crdb-01.g.apple.com -p 9090 -U yugabyte -d yugabyte |
| YCQL : bin/ycqlsh aws-us-west-2a-ybdb1d-crdb-01.g.apple.com 9091 -u cassandra |
| Data Dir : /crdb/data/yugabyte |
| Log Dir : /crdb/log/yugabyte |
| Universe UUID : 87bf301c-6baa-473c-b93c-6761dc7aeca0 |
±---------------------------------------------------------------------------------------------------------------------------------------+

NON-PROD:!:_aws-us-west-2a-ybdb1d-crdb-01:/home/crdbadmin> nc -vz aws-us-west-2a-ybdb1d-crdb-01.g.apple.com 9090
Ncat: Version 7.92 ( Ncat - Netcat for the 21st Century )
Ncat: Connected to 100.76.212.207:9090.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.

Thanks @vineet2k1 for confirming its not a firewall related issue.

As Its not related to any of the obivious reasons, we need the complete logs as Dorian had originally asked for. can you please upload the logs after collecting them using -

./bin/yugabyted collect_logs

Hi @nmalladi uploaded the logs - yugabyted-2025-02-11-04h36m41.096698s.tar.gz - Google Drive

Hi @vineet2k1, can you send the steps that you used to create the certs for the node. Just for reference, here are the steps that we recommend to follow while creating certs for yugabyteDB.

We will go over the uploaded logs and get back to you.

Also for us replicating it internally, can you please share the steps used for creating the custom certs?

Thanks,
Nikhil

hi @nmalladi we have a internal way to generate the certs. The certs seems ok, please let me know if anything else need to verify.

NON-PROD:!:_aws-us-west-2a-ybdb1d-crdb-01:/home/crdbadmin/ybdb1d/certs> openssl verify ca.crt node.aws-us-west-2a-ybdb1d-crdb-01.g.apple.com.crt
ca.crt: OK
node.aws-us-west-2a-ybdb1d-crdb-01.g.apple.com.crt: OK

@vineet2k1 The logs you’ve uploaded don’t seem to have all the required files.

The yugabyted logs only have this information. I believe you didn’t specify the --base_dir=$HOME/yb_open_binaries when running collect_logs?

[yugabyted status] 2025-01-31 23:37:28,317 INFO:  | 0.0s | Running yugabyted command: '/home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted status'
[yugabyted status] 2025-01-31 23:37:28,317 INFO:  | 0.0s | cmd = status using config file: /home/crdbadmin/var/conf/yugabyted.conf
[yugabyted status] 2025-01-31 23:37:28,317 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file openssl_proxy.sh
[yugabyted status] 2025-01-31 23:37:28,317 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file yb-admin
[yugabyted status] 2025-01-31 23:37:28,317 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file yb-ts-cli
[yugabyted admin_operation] 2025-02-04 18:30:49,799 INFO:  | 0.0s | Running yugabyted command: '/home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted configure admin_operation --command get_universe_config'
[yugabyted admin_operation] 2025-02-04 18:30:49,799 INFO:  | 0.0s | cmd = admin_operation using config file: /home/crdbadmin/var/conf/yugabyted.conf
[yugabyted admin_operation] 2025-02-04 18:30:49,799 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file openssl_proxy.sh
[yugabyted admin_operation] 2025-02-04 18:30:49,799 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file yb-admin
[yugabyted admin_operation] 2025-02-04 18:30:49,799 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file yb-ts-cli
[yugabyted admin_operation] 2025-02-04 18:30:49,800 ERROR:  | 0.0s | Traceback (most recent call last):
  File "/home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted", line 8439, in run
    args.func()
  File "/home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted", line 2574, in configure_admin_operation
    master_addrs = ",".join(self.get_all_masters())
  File "/home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted", line 4221, in get_all_masters
    raise RetryableError()
RetryableError

For more information, check the logs in /home/crdbadmin/var/logs
[yugabyted collect_logs] 2025-02-07 17:24:35,765 INFO:  | 0.0s | Running yugabyted command: '/home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted collect_logs'
[yugabyted collect_logs] 2025-02-07 17:24:35,765 INFO:  | 0.0s | cmd = collect_logs using config file: /home/crdbadmin/var/conf/yugabyted.conf
[yugabyted collect_logs] 2025-02-07 17:24:35,765 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file openssl_proxy.sh
[yugabyted collect_logs] 2025-02-07 17:24:35,765 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file yb-admin
[yugabyted collect_logs] 2025-02-07 17:24:35,765 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file yb-ts-cli
[yugabyted collect_logs] 2025-02-07 17:24:35,767 INFO:  | 0.0s | 
+-----------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                      e[1me[32myugabytede[0me[0m                                                                      |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| e[33mStatuse[0m                     : Logs collected successfully.                                                                                           |
| e[33mCollected logs pathe[0m        : Collected logs can be found at: /home/crdbadmin/yugabyte_collected_logs/yugabyted-2025-02-07-17h24m35.765924s.tar.gz   |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+

[yugabyted collect_logs] 2025-02-11 04:36:41,096 INFO:  | 0.0s | Running yugabyted command: '/home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted collect_logs'
[yugabyted collect_logs] 2025-02-11 04:36:41,096 INFO:  | 0.0s | cmd = collect_logs using config file: /home/crdbadmin/var/conf/yugabyted.conf
[yugabyted collect_logs] 2025-02-11 04:36:41,096 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file openssl_proxy.sh
[yugabyted collect_logs] 2025-02-11 04:36:41,096 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file yb-admin
[yugabyted collect_logs] 2025-02-11 04:36:41,096 INFO:  | 0.0s | Found directory /home/crdbadmin/yb_open_binaries/yugabyte-2.25.0.0/bin for file yb-ts-cli

We want the logs after you execute the below command -

$HOME/yb_open_binaries/yugabyte-2.25.0.0/bin/yugabyted start \
--config $HOME/yugabyte_start.conf \
--advertise_address=$(hostname) \
--cloud_location=aws.us-west-2.us-west-2a \
--base_dir=$HOME/yb_open_binaries \
--fault_tolerance=region
–Conf file

Please send us the logs from the failed node using the base_dir $HOME/yb_open_binaries.

./bin/yugabyted collect_logs --base_dir=$HOME/yb_open_binaries

Thanks

Sure @nmalladi done - yugabyted-2025-02-11-22h15m10.982035s.tar.gz - Google Drive

1 Like

@vineet2k1

I see in the tserver logs, postgres process is repeatedly getting terminated with the following error -

2025-02-11 22:14:27.334 UTC [2318292] FATAL:  private key file "/home/xxxx/xxxx/certs/node.aws-us-west-2a-ybdb1d-crdb-01.g.xxxx.com.key" has group or world access
2025-02-11 22:14:27.334 UTC [2318292] DETAIL:  File must have permissions u=rw (0600) or less if owned by the database user, or permissions u=rw,g=r (0640) or less if owned by root.
2025-02-11 22:14:27.334 UTC [2318292] LOG:  database system is shut down

It looks like the issue is with the permissions on the key file in the certs directory. Can you update the permissions of all the files in the cert directory to 0600 and restart the yugabyted node?

Thanks,
Nikhil

1 Like

Below are the permissions of the cert files that are created by default with yugabyted start --secure command -

nikhil@dev-server-nikhil-new:/mnt/disks/disk2/yugabyted-base-dir/certs$ ls -la
total 24
drwxrwxr-x 2 nikhil nikhil 4096 Feb 11 22:49 .
drwxrwxr-x 7 nikhil nikhil 4096 Feb 11 22:49 ..
-rw-rw-r-- 1 nikhil nikhil 1180 Feb 11 22:49 ca.crt
-rw-rw-r-- 1 nikhil nikhil 4167 Feb 11 22:49 node.10.151.0.160.crt
-r-------- 1 nikhil nikhil 1704 Feb 11 22:49 node.10.151.0.160.key
nikhil@dev-server-nikhil-new:/mnt/disks/disk2/yugabyted-base-dir/certs$ stat -c %a ca.crt
664
nikhil@dev-server-nikhil-new:/mnt/disks/disk2/yugabyted-base-dir/certs$ stat -c %a node.10.151.0.160.crt
664
nikhil@dev-server-nikhil-new:/mnt/disks/disk2/yugabyted-base-dir/certs$ stat -c %a node.10.151.0.160.key
400

Thanks a lot @nmalladi it worked, looks like the permissions issue only. I have 2 more questions:

  1. What is difference in flags secure & insecure.
  2. How to enable the mTLS with the encryption