Application Tutorials

Simple and Robust PostgreSQL Backups using PgBackWeb on nmaas

PgBackWeb is an open-source web application which can perform backups and restores on PostgreSQL databases. Backup files can be stored either locally (on the same host where PgBackWeb is running) or remotely, on any S3 compatible cloud storage.

PgBackWeb is available in the nmaas application catalogue and can be deployed on the central vNOC nmaas instance or on any self-hosted nmaas deployment.

In this tutorial we will go over the steps of deploying and configuring PgBackWeb on nmaas so that it can perform scheduled database backups of remote PostgreSQL databases. We will also see how a different tool from the application catalogue - Healthchecks can be used in tandem with PgBackWeb so that notifications can be sent if a backup hasn't been completed in the regular timeframe. Finally, we will perform a test restore of the database backup on an nmaas hosted instance of PostgreSQL, which can help you quickly recover your data and resume operations in case of a failure of the remote PostgreSQL cluster.

Deploying PgBackWeb on nmaas

To deploy PgBackWeb on nmaas the following steps need to be performed:

1. Choose PgBackWeb from the application catalogue
2. Subscribe to the application
3. Deploy the application using the application deployment wizard.
4. Create an admin account directly on PgBackWeb using the initial setup screen.

During the deployment, the nmaas deployment wizard asks only for the desired storage space to be allocated. The default value of 5Gi is enough in most cases, except when backing up very large remote databases.

By clicking on the Apply configuration button, the deployment will proceed and PgBackWeb will be available for access in a couple of moments via the Actions -> Access button.

As this is the first time PgBackWeb is being accessed, a new user form will appear.

Once logged in, we can start adding the PostgreSQL servers and databases that we would like to backup.

Configuring PgBackWeb for Backups of Remote PostgreSQL Databases

The steps that need to be performed in this part are:

1. Add the database using the Databases page.
2. Optionally add an S3 destination for backups storage using the Destinations page. If no destination is added, backups will be stored locally (on the same host where PgBackWeb) is deployed.
3. Configure scheduled backup tasks using Backup tasks page.
4. Performing the backup for the first time.

Adding Databases to PgBackWeb

To backup a remote database using PgBackWeb, the first step is to add it in the Databases section which can be accessed using the left-hand menu.

The Add database button opens a pop-up dialog, asking for the connection string to the database.

After filling out the connection details, the connection can be tested using the Test connection button. In case no issues are detected, the configuration can be saved using Add database. PgBackWeb does periodic health checks (every 10 minutes) on the database and will mark the database as down in case of any issues.

Configuring a Backup Schedule

Scheduled backups can be configured in the Backup tasks section from the left-hand menu.

The following information needs to be provided:

• Name = arbitrary name for the backup task
• Database = what database to backup. The database must first be added to PgBackWeb, as described in the previous subsection.
• Local backup = whether to do a local backup or upload the dump files to a remote storage.
• Cron expression = cron expression specifying the backup schedule.
• Time zone = time zone to use for the cron expression.
• Destination directory = in case a local backup is selected, where to store the backups. On nmaas, the /backups directory is available for backups storage.
• Retention days = how many days backups are kept before deleting them.

Performing the Backup for the First Time

From the Backup tasks overview page, the configured backup task can be executed in an ad-hoc fashion using the Run backup now option.

The Executions section from the left-hand menu shows details about each backup execution, either ad-hoc or as part of a scheduled run.

Monitoring for Backup Failures using Healthchecks

PgBackWeb does not have a built-in way to send notifications in case of backup failures. However, it can send webhooks to remote systems. We can leverage Healthchecks, another application from the nmaas catalogue, to build a notification system for PgBackWeb. The scenario is as follows:

1. Healthchecks generates a unique URL where it expects that a request will be sent every X minutes (the same schedule as configured in PgBackWeb previously).
2. PgBackWeb sends a webhook for each successful backup run to the Healthchecks URL.
3. Healthchecks sends a notification if no request is received on the configured URL (meaning that the backup either was not executed or was not successful).

Deploying Healthchecks on nmaas

Healthchecks can be deployed in a similar manner to any other application on nmaas - first a subscribe to the application is needed, and then further configuration can be done using the deployment wizard.

The deployment wizard for Healthchecks expects the following information:

• Email address for the initial admin user account
• Password for the initial admin user account
• Open registration = whether the sign-up feature will be enabled so that anyone can register a new account.
• Access options = whether the Healthchecks interface will be publicly accessible via the Internet or it will be constrained to the client-access VPN. As we will be using Healthchecks only in tandem with PgBackWeb, in this case it is recommended to simply select VPN.
• Allocated storage space = storage space to allocate to the application, the default 1Gi is sufficient in most cases.

Healthchecks has the concept of Projects in which you can organize similar checks. After deployment, by default, Healthchecks will create a new project named after your username.

Notification channels can be configured using the Integrations section, and by default email notifications are already setup for the default project but will need to be manually setup for all subsequent ones.

We will create a new project named PgBackWeb Backups by using the New Project... button on the homepage. As no integrations are configured in new projects by default, there will be a red exclamation mark next to the Integrations button in the navbar. We can configure email notifications simply by entering the recipient's email address. SMTP settings are configured by default for all Healthchecks instances deployed on nmaas.

Finally, we can add a new check from the Checks section.

Once the check is created, the unique URL will be shown and can be copied directly just by clicking it.

We are now ready to configure the webhook in PgBackWeb.

Webhook Configuration in PgBackWeb

In PgBackWeb, in the Webhooks section a new webhook can be added using the Create webhook button.

The following information needs to be provided:

• Name = an arbitrary name to identify the webhook entry.
• Event type = select Execution success.
• Backup targets = select the backup targets for which the webhook will fire. It is recommended to configure one webhook per backup target.
• Activate webhook = Yes.
• URL = the URL copied from Healthchecks.
• Method = GET, as this is supported by Healthchecks.
• Headers = can be left empty.
• Body = can be left empty.

You can perform a test run of the webhook using the three dots next to its name and choosing Run webhook now.

A corresponding entry should immediately become visible in the webhook execution history page in Healthchecks.

Performing Restores on a PostgreSQL Instance Deployed on nmaas

In case of a data corruption any backup can be restored to the same database server from which the initial backup was obtained using the Executions section and choosing Restore execution.

However, in case the remote PostgreSQL server becomes completely inaccessible, a quick way to restore the data is to create a PostgreSQL instance on nmaas and restore the backup there. As this instance can also be accessed by any service hosted remotely via the site-to-site tunnel, such an approach can be seen as an effective way to continue operations while the original server is recovered.

PostgreSQL can be deployed as any other application on top of nmaas. During the deployment process, the following parameters need to be provided:

• Root password = the password for the postgres user.
• Database user = the username for the default user.
• Database password = the password for the default user.
• Database name = the database name of the initial database.

Once deployed, access details are available from the Actions -> Access pop-up. For remote access, the External access options parameters should be used, while for accessing PostgreSQL from other applications deployed on nmaas, the Local (in-cluster) access options are relevant.

Back in PgBackWeb, in the Restore backup execution pop-up we can choose as Backup to the Other database option and provide the connection string to the newly deployed PostgreSQL instance.

The restore progress can be checked from the Restorations section.

The contents of the restored database can be checked with another application already available on nmaas - Adminer. Adminer is a web-based application which can be used to manage PostgreSQL and various other databases.

Conclusion

In this tutorial we have performed the following steps:

1. Deployed PgBackWeb.
2. Configured PgBackWeb to perform scheduled backups of a remote database.
3. Deployed Healthchecks.
4. Configured Healthchecks with a new unique monitoring URL and an email notifications channel.
5. Configured PgBackWeb to execute a webhook upon each successful backup execution.
6. Deployed PostgreSQL on nmaas.
7. Restored the initial backup performed by PgBackWeb of the remote database to the PostgreSQL instance on nmaas.

[ #004 ] NMaaS-101 - "Prometheus"

In article #001 and #002 you essentially created a NMaaS domain and enabled connectivity between your dedicated and isolated domain to your out of band management equipment network.

Overview

In this post, we are going to deploy an interesting and popular Metric collector in the micro-service world: Prometheus. For those who would like an introduction to Prometheus, please refer to this post from the RARE project blog.

Article objective

This is done in 2 steps:

• Prometheus application deployment via the NMaaS portal
• Prometheus configuration specific to RARE domain

Diagram

NMaaS portal: Oxidized

[#003] - Cookbook

1
2
3
4
5
6
7

git clone ssh://[email protected]/groups-rare/rare-prometheus-382.git
< enter my SSH credientials ... >
cd rare-prometheus-382
╭─[05/6/21|11:18:47][email protected] ~/rare-prometheus-382  ‹master›
╰─➤  ls -l
total 8
-rw-r--r--  1 loui  staff  297 May  6 11:17 prometheus.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

global:
    scrape_interval: 15s
    evaluation_interval: 30s
alerting:
    alertmanagers:
        - static_configs:
            - targets:
rule_files:
scrape_configs:
    - job_name: 'router'
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
    - targets: ['192.168.0.1:9001','192.168.0.2:9001']
        labels:

Verification

Conclusion

In this article you:

• You have deployed a powerful and flexible metric collector for your organisation
• Prometheus uses PUSH model similar to SNMP so every scrape minutes it will interrogate all the configured agents.
• You have learned how to apply specific configurations to it in order to match your requirements
• In this example, we used RARE/freeRtr prometheus agent whose configuration is described here. In your case, you agent will have its own different configuration (different IP, port, job name and metrics )

[ #003 ] NMaaS-101 - "My name is Oxidized and I'm a nifty configuration management tool"

In article #001 and #002 you essentially created a NMaaS domain and enabled connectivity between your dedicated and isolated domain to your out of band management equipment network.

Overview

We are going to deploy our first NMaaS service for your organisation: Oxidized

Article objective

This is done in 2 steps:

• Oxidized application deployment via the NMaaS portal
• Oxidized configuration specific to RARE domain

Diagram

NMaaS portal: Oxidized

[#003] - Cookbook

1
2
3
4
5

cd base
ls -l
total 16
-rw-r--r--  1 loui  staff  734 Jul 30 11:12 config
-rw-r--r--  1 loui  staff  141 Jul 30 11:12 router.db

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

---
username: rare
password: rare
model: rare
interval: 600
use_syslog: false
debug: false
threads: 30
timeout: 20
retries: 3
prompt: !ruby/regexp /([\w.@-]+[#>]\s?)$/
rest: 0.0.0.0:8888
vars: {}
groups:
    wedge-bf100-32x:
        vars:
            ssh_port: 2001
pid: "/storage/pid"
input:
    default: ssh
    debug: false
    ssh:
        secure: false
output:
    default: git
    file:
    directory: "/storage/configs"
git:
    single_repo: true
    user: oxidized
    email: [email protected]
    repo: "/storage/oxidized.git"
source:
    default: csv
    csv:
        file: "/root/.config/oxidized/router.db"
        delimiter: !ruby/regexp /:/
        map:
            name: 0
            model: 1
            group: 2
model_map:
    rare: rare
    cisco: ios
    juniper: junos

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

class RARE < Oxidized::Model
    prompt /([\w.@()-]+[#>]\s?)$/
    #prompt /^([\w.@()-]+[#>]\s?)$/
    comment '! '
    cmd :all do |cfg|
        # cfg.gsub! /\cH+\s{8}/, '' # example how to handle pager cfg.gsub! /\cH+/, '' # example how to handle pager get rid of errors for commands that don't work on some devices
        cfg.gsub! /^% Invalid input detected at '\^' marker\.$|^\s+\^$/, ''
        cfg.cut_both
end
cmd :secret do |cfg|
    cfg.gsub! /^(snmp-server community).*/, '\\1 <configuration removed>'
    cfg.gsub! /^(snmp-server host \S+( vrf \S+)?( version (1|2c|3))?)\s+\S+((\s+\S*)*)\s*/, '\\1 <secret hidden> \\5'
    cfg.gsub! /^(username .+ (password|secret) \d) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(enable (password|secret)( level \d+)? \d) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+(?:password|secret)) (?:\d )?\S+/, '\\1 <secret hidden>'
    cfg.gsub! /^(.*wpa-psk ascii \d) (\S+)/, '\\1 <secret hidden>'
    cfg.gsub! /^(.*key 7) (\d.+)/, '\\1 <secret hidden>'
    cfg.gsub! /^(tacacs-server (.+ )?key) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(crypto isakmp key) (\S+) (.*)/, '\\1 <secret hidden> \\3'
    cfg.gsub! /^(\s+ip ospf message-digest-key \d+ md5) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+ip ospf authentication-key) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+neighbor \S+ password) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+vrrp \d+ authentication text) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+standby \d+ authentication) .{1,8}$/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+standby \d+ authentication md5 key-string) .+?( timeout \d+)?$/, '\\1 <secret hidden> \\2'
    cfg.gsub! /^(\s+key-string) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^((tacacs|radius) server [^\n]+\n(\s+[^\n]+\n)*\s+key) [^\n]+$/m, '\1 <secret hidden>'
    cfg
end
cmd 'show platform' do |cfg|
    comment "TEST: show platform"
    comments = []
    comments << cfg.lines.first
    lines = cfg.lines
    lines.each_with_index do |line, i|
        if line !~ /^mem:|^uptime:/
            comments << line.strip!
        end
    end
    comments << "\n"
    comment comments.join "\n"
end
cmd 'show interfaces description' do |cfg|
    comment cfg
end
cmd 'show running-config' do |cfg|
    cfg = cfg.each_line.to_a[3..-1]
    cfg = cfg.reject { |line| line.match /^ntp clock-period / }.join
    cfg.gsub! /^Current configuration : [^\n]*\n/, ''
    cfg.gsub! /^ tunnel mpls traffic-eng bandwidth[^\n]*\n*(
              (?: [^\n]*\n*)*
              tunnel mpls traffic-eng auto-bw)/mx, '\1'
    cfg
end
cfg :telnet do
    username /^Username:/i
    password /^Password:/i
end
cfg :telnet, :ssh do
    # preferred way to handle additional passwords
    post_login do
      if vars(:enable) == true
         cmd "enable"
      elsif vars(:enable)
        cmd "enable", /^[pP]assword:/
        cmd vars(:enable)
        end
    end
    post_login 'terminal length 0'
    post_login 'terminal width 0'
    pre_logout 'exit'
    end
end

1
2
3
4
5

cd model
ls -l
total 16
-rw-r--r--  1 loui  staff  2977 Jul 30 11:13 rare.rb
-rw-r--r--  1 loui  staff    69 Jul 30 11:10 readme.txt

Verification

Conclusion

In this article you:

• You have deployed a powerful CMDB software for your organisation
• You have learned how to apply specific configurations to it in order to match your requirements