FileCloud’s database service offers a flexible scheme to store large data sets, which can be configured for data redundancy and high availability. Note: this blog post is intended for system and IT administrators who manage FileCloud Server environments. Geographic Redundancy The database service used by FileCloud uses a replica set to support geographic redundancy and […]
FileCloud's database service offers a flexible scheme to store large data sets, which can be configured for data redundancy and high availability.
Note: this blog post is intended for system and IT administrators who manage FileCloud Server environments.
The database service used by FileCloud uses a replica set to support geographic redundancy and increase data availability. Having multiple copies of data on different database servers provides a level of fault tolerance against the loss of a single database server.
The replica set should contain a minimum of three members with only one server being primary and the other two being secondary. The primary server will receive all write operations and then data is replicated to the secondary servers.
If the primary member fails, a new primary is elected from the other two active secondary servers; this process is done by an internal election process. This election process requires three voting members with a priority value.
This blog post will outline how to use voting ability and priority values to control which server is designated the primary. The post will also review the steps necessary to set up the server to perform write operations, which will be necessary if you want to geographically distribute your replica set. Doing so will provide geographic redundancy for your FileCloud site.
A typical FileCloud single location architecture contains an application server (Apache and other services), MongoDB database servers, and a storage server. FileCloud can be deployed for geographic redundancy if the storage system provides geographic replication capabilities.
While FileCloud has its own geographic replication solution with the help of FileCloud ServerLink, this might not be an option for use cases where Network Shares are being used or if the replication process overhead over HTTPS needs to be minimized.
In similar use cases, FileCloud can be deployed across multiple locations by adding additional application servers and database servers, provided storage is replicated on the customer side. The below architecture is an example of such a deployment where each component (e.g., application, database, and storage server) is located across multiple geographic locations.
This architecture can help with situations where a customer’s primary location is not accessible due to power outages, network interruptions, natural disasters, or maintenance.
Similar architecture can be achieved with three (3) application servers and three (3) database servers in a replica set configuration, with each of the application and database servers located in each location. In such an environment, whenever the application is operated from a specific location, it is important that the database write and read operations are also done from the same location for optimum performance and user experience.
Replica set configurations with custom priority values can be used to forcefully assign a primary replica set member in this case.
Let’s assume we have three locations: Location A, Location B, and Location C. Your primary need is to serve the application and database operations from Location A.
Primary members of the database service can change whenever connectivity is lost between the primary and other database members; the members who can connect with each other will vote to designate a new primary.
Assuming a scenario where the Location A database server loses connectivity for a short amount of time between the other two database servers, either the Location B or Location C database server will become primary; they will continue to serve in this state unless there is another election process involved.
With our use case outlined above, we want Location A to serve the application and database operations. Consequently, we want to avoid Location A being replaced by another server in the event of an outage.
To accomplish this, we can set the priority for the Location A server to be high, so that whenever the connectivity between the servers is normal, the replica set members continue to call elections until the highest priority member available becomes primary.
This can be configured by connecting to a replica set Mongo Shell and then setting the priority of each member based on their identity order in the cluster status.
An example set of configurations is as follows:
cfg = rs.conf()
cfg.members[0].priority = 3
cfg.members[1].priority = 2
cfg.members[2].priority = 1
rs.reconfig(cfg)
Here, member 0 is Location A, member 1 is Location B, and member 2 is Location C.
While a minimum of three server replica set clusters across three different regions will provide a minimum level of fault tolerance, it does introduce a risk of network drops and connection issues that may occur across multiple geographical locations.
Since the FileCloud database service needs at least two servers for the election process and active write operations, connection drops across three locations can cause application disruption. To tackle similar issues, this architecture can be stabilized by adding more database servers to the replica set for more fault tolerance.
For environments with minimum or no network disruption or connectivity issues, this is not a requirement or need. However, considering situations where the three locations cannot connect to each other, the replica set would not be able to perform write operations. Without these operations, applications may be disrupted due to majority availability. Adding an additional database server in each location will provide more fault tolerance with more high-priority members available in each location.
In a situation where each location cannot talk to the other two locations, the replica set provides flexibility by manually forcing the primary write operation from a specific location via priority values and voting abilities of each member.
To set priority values and voting abilities, non-voting members and zero-priority members must be designated in the replica set configuration. In our example, only three servers have the voting ability and priority set (per the configuration depicted below).
In the above configuration, one of the servers in Location A will serve as primary, which performs read-write operations. The other five servers will have a copy of replicated data. Even if there is a connection drop between Location A and Location B, the replica set will have a majority availability from Location A.
In case of any disasters, the replica set cluster can be reconfigured to have members with voting ability and priority for a new election process, thereby serving requests from a new primary.
A similar configuration can be achieved by following the configuration set described below. By default, all members will have voting ability, so we don’t need to explicitly set these abilities for the first three servers.
cfg = rs.conf()
cfg.members[0].priority = 3
cfg.members[1].priority = 3
cfg.members[2].priority = 2
rs.reconfig(cfg)
For the rest of the servers, we need to remove the priority and voting ability, one by one, per the following example:
To remove voting and priority for the second server in Location B:
cfg = rs.conf()
cfg.members[3].priority = 0
cfg.members[3].votes = 0
rs.reconfig(cfg)
To remove voting and priority for the first server in Location C:
cfg = rs.conf()
cfg.members[4].priority = 0
cfg.members[4].votes = 0
rs.reconfig(cfg)
To remove voting and priority for the second server in Location C:
cfg = rs.conf()
cfg.members[5].priority = 0
cfg.members[5].votes = 0
rs.reconfig(cfg)
If Location A is not available, and if the replica set needs to be reconfigured with the rest of the available servers, it can be done by reconfiguring the cluster by setting two more voting members and priority according to preference, one by one.
To enable voting and priority for the second server in Location B:
cfg = rs.conf()
cfg.members[3].votes = 1
cfg.members[3].priority = 2
rs.reconfig(cfg, {force : true})
To enable voting and priority for the first server in Location C:
cfg = rs.conf()
cfg.members[4].votes = 1
cfg.members[4].priority = 1
rs.reconfig(cfg, {force : true})
Here the cluster will function with two servers from Location B (i.e., member[2] and member[3]) and one server from Location C, identified as member[4], since there is a majority availability.
This post is based on specific assumptions; however, database service replica sets can be configured based on different scenarios and use cases based on priority.
This database service enables admins to set up a FileCloud environment with maximum redundancy and availability across multiple geographical locations. Thus admins are able to overcome network interruptions and disasters, resulting in reliable performance and an improved user experience.
To learn more about how to configure FileCloud for geographic redundancy and high availability, check out our documentation or reach out to our support team.
Article written by Nejinvarghese Koshy, Senior Technical Support Engineer
Edited by Katie Gerhardt, Jr. Product Marketing Manager
