How to configure a Transaction Replication between two Azure SQL Managed Instances

Before to start with the steps required for configuring a new Transactional Replication. I want to briefly
explain you why we should consider to use Transactional Replication feature? I think there are plenty of
good reasons, but in my opinion the main reason or advantage associated to use Transactional
replication is the capacity of replicating data from a table in Azure SQL Managed Instance (or SQL Server
on premises) toward tables placed on remote databases without necessity of managing all the tables of
a Database and pick up only the few tables we want to get.

I think that is very common around companies the possibility of having a Datawarehouse which has
process to consume data of the main Databases. However, the direct consumption on the operational
Databases would have a negative impact that can be translates into performance penalties on the
“customer experience” in favor of “getting the data” for the Datawarehouse. On the base of this
scenario, I used to recommend the implementation of Transactional Replication as a valid and feasible
feature, specially when we do not have a real time expectation and can perfectly rely on a short delivery
latency.

I have seen in the past the use of different technologies as log shipping where the read queries are
executed against the log server, but to be honest, I really encourage to use a proper tool as Transaction
replication which fits in more suitable way to your requirements.

The scenario that will be covered through this article is the publisher and the distributor hosted in the
same Azure Managed Instance which would be a role as publisher and distributor and another Azure
Managed Instance which will be the subscriber. The following image summarizes the architecture:

Source: Microsoft

The official Microsoft documentation states as mandatory the following conditions:

 That the publisher managed instance is on the same virtual network as the distributor and the
subscriber, or VPN gateways have been configured between the virtual networks of all three
entities.
 Connectivity uses SQL Authentication between replication participants.
  An Azure storage account share for the replication working directory.
 Port 445 (TCP outbound) is open in the security rules of NSG for the managed instances to
access the Azure file share.

More information can be found on this link:

Defining the pieces involves

As every cuisine recipe, here the ingredients (components) that we must use for the successfully
implementation of our new Transaction Replication model.

 Define a resource group: The proposed name is TransactionalReplication

 Create a new Virtual Network (VNet), it is extremely important to remind that ALL the new
Azure SQL Managed Instance have to be hosted in the same VNet. The proposed name is:
VNetTransReplication
 Create two new Azure SQL Managed Instances. They must be hosted in the VNet defined
previously. The suggested names are : mi-tr-publisher and mi-tr-suscriber.
 Create an Azure storage account and more specifically a file share
 Create a new Azure VM for connecting new Managed Instances

I do not want to go in details about how to configure and deploy a VNet and two Azure SQL Managed
Instances, for this reason I will be limiting to only sharing you the main screens after each configuration.

Resource Group

Virtual Network

You have the option to define by yourself the Subnet, it is so important than both Azure SQL Managed
Instances share the same subnet. It is mandatory that during the provisioning stage you indicate the
subnet that you have previously created or allow it being defined during the first managed instance and
reuse for the second one. The following image shows you how both MI are using the same VNet and
subnet
Azure Managed Instance

If you want to read a step-by-step guide, I invite you to check this article: https://bit.ly/3s9Q8sc . The
final Azure SQL Managed Instances looks in this way
Azure VM for connecting new Managed Instances

For having the opportunity to connect into your new managed instances, you could deploy a new Azure
VM which should be placed in a separate subnet inside the same VNet where your managed instances
are located.

Here an example about the internal configuration setup inside the VM

You can find a detailed explanation in this link:

https://docs.microsoft.com/en-us/azure/azure-sql/managed-instance/connect-vm-instance-configure

Create an Azure storage account

From this point in advance, I will be teaching you every step to configure the key components that are
part of the required pipeline for the building of a transaction replication solution. An Azure storage
account is a required component. The official definition of Microsoft about an azure storage account is
the following:

“An Azure storage account contains all of your Azure Storage data objects: blobs, file shares, queues,
tables, and disks. The storage account provides a unique namespace for your Azure Storage data that's
accessible from anywhere in the world over HTTP or HTTPS. Data in your storage account is durable and
highly available, secure, and massively scalable.”

Why do we need an azure storage account? The answer is simple, because the Transaction Replication
technology trust on a specific file shared (that belongs to the storage account) for push and pull the
internal files generated as part of the replication structure, it means, the bcp, idx, pre, sch. Observe the
following images

Here, the first step consists in define our new storage account. You have to go Services and write
storage account and fill the boxes associated
Respect to the Advanced tab, for the simplicity of this article I will keep on the default values, equal than
Networking. This last one, perf default you can allow the access from all networks using the Public
Endpoint. However, is highly recommended to use a Private Endpoint. However, it would be required
to define a new Subnet under our VNet (VNetTransReplication).
Once that you have completed and deployed the new Storage account, you should define the new File
Shares which should be used by Transactional Replication.

After click in File shares, you should click on File Share button and it displays a window like this:
Pay attention to the SMB protocol and message associated to port 445. Let us to return to this point in
the following section. After that it has been created, you should see an screen like this

It is important to taking account the Quota and the Tier. In our case, the optimization of this file share
intended for transactions, which is the target for the features that we are going to deploy. We have to
prepare some values that we are going to use in the configuration of publish/distributor. The first is the
name of the new file share, copying the original value on the properties

We have to transform in the standard format for shared folders, it is:

\\replstoragemi.file.core.windows.net\replstoragemishare

Finally, we need to get the Default Endpoints Protocol, simply return to the storage account and click in
Access Keys option under Security + Networking and show keys. Copy the Connection String
With this information carefully stored, we only need to confirms a last step, related to the security.
Remember a message about SMB protocol and port 445, it means that we have to add (in case does not
exists) a rule at NSG level to allow the use of this protocol.

The first thing that we have to do is identify which is the NSG (Network Security Group) associated to
our Subnet where the Azure Managed Instances resides. For this case, the simple way is to go Resource
Group and looking for NSG objects, inside of the one which is related to the name of the managed
instance and click and go to Subnets and verify is the related to the Managed Instances.

Inside the Outbound security rules, create a new one where the port must be 445. Here an example
The name for the new outbound security rule is MI_Access_445_SMB_ReplicationWorkingDirectory.
The final configuration should look like this screen:

Creating a publish Database

From now to the end of this article, I am going to use a very simple example based on a main database
in the publisher and two tables which should be replicated in a subscriber database. Let me start to
define a publish Database
 1. USE [master]
2. GO
3.
4. CREATE DATABASE [MyTranReplDB_PUB]
5. GO
6.
7. USE [MyTranReplDB_PUB]
8. GO
9. CREATE TABLE ReplicaData (
10. ID INT NOT NULL PRIMARY KEY,
11. DataDescription VARCHAR(100) NOT NULL,
12. CreateDate DATETIME2 NOT NULL DEFAULT getdate()
13. )
14. GO
15.
16.
17. USE [MyTranReplDB_PUB]
18. GO
19.
20. INSERT INTO ReplicaData (ID, DataDescription) VALUES (1, 'publisher data')
21. INSERT INTO ReplicaData (ID, DataDescription) VALUES (2, 'publisher data')
22. INSERT INTO ReplicaData (ID, DataDescription) VALUES (3, 'publisher data')
23. INSERT INTO ReplicaData (ID, DataDescription) VALUES (4, 'publisher data')
24. INSERT INTO ReplicaData (ID, DataDescription) VALUES (5, 'publisher data')
25. GO
26.

Creating a subscription Database

At this case, we only need to build the target table which should have the same structure than in
replication server.

1. USE [master]
2. GO
3.
4. CREATE DATABASE [MyTranReplDB_SUB]
5. GO
6.
7. USE [MyTranReplDB_SUB]
8. GO
9. CREATE TABLE ReplicaData (
10. ID INT NOT NULL PRIMARY KEY,
11. c1 VARCHAR(100) NOT NULL,
12. dt1 DATETIME NOT NULL DEFAULT getdate()
13. )
14. GO
15.

Setting up the Distribution

It is required to connect into your mi-tr-publisher and using a client as SQL Server Management Studio
or Azure Data Studio and execute the following T-SQL code for configuring your distribution database

1. USE [master]
2. GO
3.
4. EXEC sp_adddistributor @distributor = @@ServerName;
5. EXEC sp_adddistributiondb @database = N'distribution';
6. GO
7.

Configure publisher to use distributor

It is important to remember our option is to have the publisher and distributor on mi-tr-publisher.
Before to execute the query, you must change the query execution to SQLCMD mode and execute it.
This code would register a new distributor with our publisher, configures a local publisher on the
managed instance, adds a linked server, and creates a set of jobs for the SQL Server Agent.

1. :setvar username VM-Admin

2. :setvar password 7*GmxPLzex3+;jpL
3. :setvar file_storage "\\replstoragemi.file.core.windows.net\replstoragemishare"
4. -- example: file_storage "\\replstorage.file.core.windows.net\replshare"
5. :setvar file_storage_key
"DefaultEndpointsProtocol=https;AccountName=replstoragemi;AccountKey=jFggcqBGw5qzUUb
V7wRJYDP61gzzbnOI79T//
ba1AbRGczXwWYBHymd3Rws3pT1R5B3mF9EiS2PXhKeZ8ZWs8Q==;EndpointSuffix=core.windows.net"
6. -- example: file_storage_key
"DefaultEndpointsProtocol=https;AccountName=replstorage;AccountKey=dYT5hHZVu9aTgIteG
fpYE64cfis0mpKTmmc8+EP53GxuRg6TCwe5eTYWrQM4AmQSG5lb3OBskhg==;EndpointSuffix=core.win
dows.net"
7.
8. USE [master]
9. EXEC sp_adddistpublisher
10. @publisher = @@ServerName,
11. @distribution_db = N'distribution',
12. @security_mode = 0,
13. @login = N'$(username)',
14. @password = N'$(password)',
15. @working_directory = N'$(file_storage)',
16. @storage_connection_string = N'$(file_storage_key)';

It is important to highlight than the previous script is in charge of building the pipeline for enabling the
Transaction Replication. As you can see, we are passing inside of the script parameters which values
corresponding to the definition of the file storage that we have defined recently.

A key element we should keep in mind is the fact of having linked servers configured as part of this
implementation, so we always should verify that as part of the Outbound Security Rules, we have one
related to the linked server outbound, here an example:
Create publication and subscriber

Inside of server mi-tr-publisher, and using SQLCMD mode, run the following T-SQL script to enable
replication for your database, and configure replication between your publisher, distributor, and
subscriber.

1. -- Set variables
2. :setvar username TransReplication
3. :setvar password uC}6Qn]xLrE%)j*\
4. :setvar source_db MyTranReplDB_PUB
5. :setvar publication_name PublishData
6. :setvar object ReplicaData
7. :setvar schema dbo
8. :setvar target_server "mi-tr-suscriber.4611b8d14acb.database.windows.net"
9. :setvar target_username TransReplication
10. :setvar target_password uC}6Qn]xLrE%)j*\
11. :setvar target_db MyTranReplDB_SUB
12.
13. -- Enable replication for our source database
14. USE [$(source_db)]
 15. EXEC sp_replicationdboption
16. @dbname = N'$(source_db)',
17. @optname = N'publish',
18. @value = N'true';
19.
20. -- Create our publication
21. EXEC sp_addpublication
22. @publication = N'$(publication_name)',
23. @status = N'active';
24.
25.
26. -- Configure the log reader agent
27. EXEC sp_changelogreader_agent
28. @publisher_security_mode = 0,
29. @publisher_login = N'$(username)',
30. @publisher_password = N'$(password)',
31. @job_login = N'$(username)',
32. @job_password = N'$(password)';
33.
34. -- Add the publication snapshot
35. EXEC sp_addpublication_snapshot
36. @publication = N'$(publication_name)',
37. @frequency_type = 1,
38. @publisher_security_mode = 0,
39. @publisher_login = N'$(username)',
40. @publisher_password = N'$(password)',
41. @job_login = N'$(username)',
42. @job_password = N'$(password)';
43.
44. -- Add the target table to the publication
45. EXEC sp_addarticle
46. @publication = N'$(publication_name)',
47. @type = N'logbased',
48. @article = N'$(object)',
49. @source_object = N'$(object)',
50. @source_owner = N'$(schema)';
51.
52. -- Add the subscriber
53. EXEC sp_addsubscription
54. @publication = N'$(publication_name)',
55. @subscriber = N'$(target_server)',
56. @destination_db = N'$(target_db)',
57. @subscription_type = N'Push';
58.
59. -- Create the push subscription agent
60. EXEC sp_addpushsubscription_agent
61. @publication = N'$(publication_name)',
62. @subscriber = N'$(target_server)',
63. @subscriber_db = N'$(target_db)',
64. @subscriber_security_mode = 0,
65. @subscriber_login = N'$(target_username)',
66. @subscriber_password = N'$(target_password)',
67. @job_login = N'$(username)',
68. @job_password = N'$(password)';
69.
70. -- Initialize the snapshot
71. EXEC sp_startpublication_snapshot
72. @publication = N'$(publication_name)';
73.

At this point, we have completed all the steps required for the configuration of a new Transaction
Replication feature. Currently, Microsoft recognizes that Azure SQL Managed Instance is experiencing
some backend issues with connectivity with the replication agents. While this issue is being addressed,
the workaround is to increase the login timeout value for the replication agents. Here, the following
script that helps on it.

1. -- Increase login timeout to 150s

2. update msdb..sysjobsteps set command = command + N' -LoginTimeout 150'
3. where subsystem in ('Distribution','LogReader','Snapshot') and command not like '%-
LoginTimeout %'

Testing our changes

Finally, the best and simple way of testing consist on connecting to both Managed Instances Server from
our Virtual Machine. Inside the mi-tr-publisher do a simple insert.

1. insert into ReplicaData(ID,DataDescription)

2. values(8,'new publish data 8')

So, we should connect to the managed instance mi-tr-suscriber and do a simple select for checking the
target table ReplicaData and verify that all the rows are present. Note: In dependence of different
factors, could be possible to have a little delay.

Here the final images

Publisher Insertion

Subscriber got replicated data

Conclusion

As you can read through this article, the implementation of Transactional Replication (TR) feature
requires a good bunch of planning around all the pieces and their respective interaction (and the
permissions involved). Respect to this feature (TR), we can state than from a performance and return of
investment (ROI) perspective is a good candidate for scenarios such as Datawarehouse and third partly
component which requires read-only queries.

It is important to notice that the secondary or subscriber Azure Managed Instance can have a basic
configuration and lower number of resources. It could increase on dependency of the business
necessity, but my experience is that we should start with a basic and humble number of resources and
increases only if it is profitable. Remember that the idea is to include as part of the
publisher/distributor/subscriber scheme the tables that we really need.

Finally, it is important to remember to monitor the effectively of the new feature implementation. So, it
is vital to keep monitoring and do not avoid to setup some rules for keeping sure and working the new
JOBS that were enable as part of the implementation of Transaction Replication.
Links

https://www.mssqltips.com/sqlservertip/4892/monitor-sql-replication-log-reader-agent-latency/

https://docs.microsoft.com/en-us/azure/azure-sql/managed-instance/replication-transactional-
overview

https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview

https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-create-file-share?tabs=azure-
portal

https://docs.microsoft.com/en-us/azure/azure-sql/managed-instance/replication-between-two-
instances-configure-tutorial

https://scomandothergeekystuff.com/2020/06/22/azure-service-endpoints-versus-azure-private-links/
comment-page-1/

https://docs.microsoft.com/en-us/azure/azure-sql/managed-instance/public-endpoint-configure

https://www.mssqltips.com/sqlservertip/3598/troubleshooting-transactional-replication-latency-issues-
in-sql-server/