Skip to content

FreePBX Remote Backups

We all (hopefully) backup our PBX's - our config files, databases, even system audio and logs, some of this can be backed up hourly, daily, or weekly, so in all likelihood, you will have at least two backup processes probably more, but where do you back them up to?

Local storage? It's fair to say nearly everyone uses this, but is that the only place you have them? What if your disks fail? A lot of good those backups are now you can't access them!

Therefore it's always a very good idea, one could say mandatory, to also keep an external or remote backup, but you need to make sure that process from start to end is secure, obviously a local external backup is ideal, for example using a Synology NAS or spare PC, where securing them is within your control, but often businesses require off-site remote backups, easy enough if you backup to a local Synology NAS which can replicate that backup to another remote located Synology NAS, or backing up via SCP to a remote server - but do you trust whoever manages the remote site? Do they run it securely? The number of AWS buckets that are unsecured is astounding, then there's the physical access concerns, who can access the remote device and your data? If it's on a third party device, the risks are even greater, you get the idea.

Remember, when creating your backup storage servers, you can use multiple storage types simultaneously.

OK, so you've got a secure remote server to store your backups on, but what about getting your data securely to it?

I felt compelled to write this article after having seen a video on Youtube from someone who is supposed to be a FreePBX guru who also does FreePBX training videos for Sangoma (Sangoma University - their online certification courses), when I was horrified to see him putting out this instructional video telling people to use FTP because (big jaw drop moment) that's what he does for his customers.


W T A F?
Look, in a perfect world nobody would care, but we live in a far from perfect world, FTP transfers everything in the clear, plain and unencrypted data streams anyone can sniff out, your username, password, the data, it's all there for the taking because you're not using a secure transport method.

Now, before the fanbois jump up and say FTPS FTPS FTPS (FTP over TLS) which is secure, that requires two critical components, first, your remote server must support it, second, and the problem here, is that FreePBX does not support it, the FTP storage option only uses plain FTP, not FTPS despite the php filesystem component FreePBX uses actually supporting SSL, but Sangoma choose not to offer or use it, otherwise it would be an acceptable secure method, quick and easy to set up, but until Sangoma fix this we need to use other methods that are secure.

SSH Storage A Better Solution

FreePBX's SSH implementation is a better option, but only if you control, lock down, and completely trust the backup server, if it's compromised, it could be trying to write and create directories anywhere it has write permissions on the filesystem, which to prevent really requires creating jails for each user, else they could login and go peeking around everywhere, not as secure but much simpler to implement, is using openssh's special command=/some/script option, preceding a key in authorized_keys which can be used to deny ssh, but allow scp, eg:

All way more work than anyone will likely bother with.

Note: In years gone by we have used rssh to restrict users to certain components, it served everyone well for over a decade, but rssh is no longer recommended for use, even its author says nobody should be using rssh, it can be exploited, and is abandonware.

Let's Get Started

To make thing easy, open two terminal windows, in one ssh into the PBX and in the other, into the backup server.

On the backup server create a user unique to the particular PBX, in our example here, I'll be using pbx123, then cd to their home directory and create the .ssh directory (don't forget to chown) and use your editor of choice to create a file in the .ssh directory called authorized_keys (note the spelling of authorized).

On the PBX we need to create the SSH keys, modern operating systems should use ed25519, if this fails, use the legacy RSA method below which although slow, if using a key size of 4096 is still secure enough today, I will use id_pbx as the filename for simplifying my examples, so id_pbx is the private key file and id_pbx.pub is the public key file.

cd /home/asterisk

ssh-keygen -q -f id_pbx -t ed25519
     (hit enter when it asks for passwords)
chown asterisk id_pbx

cat id_pbx.pub        (and copy its contents)

The Legacy and much slower RSA method if ed25519 fails, likely on CentOS prior to 7
* Important - As of OpenSSH 8.8. RSA will be disabled, so this method should be avoided, time to start planning your distro upgrade instead. But if you accept the risks...

cd /home/asterisk

ssh-keygen -q -b 4096 -f id_pbx -t rsa
     (hit enter when it asks for passwords)
chown asterisk id_pbx

cat id_pbx.pub        (and copy its contents)

Next copy the contents of your public keyfile (id_pbx.pub) into the ~/.ssh/authorized_keys file of the user pbx123 on the backup server, save and exit the file, the private key file (id_pbx) stays in /home/asterisk on the PBX.

Because keys need to be accepted, on your PBX, ssh into the backup server as the user pbx123 and accept the keys when prompted, this step is important, if you omit it, your transfers (automatic ssh login) will fail, finally, logout, and then back in to verify the login, this time you should not be prompted, if you are, especially for a password, check the messages on screen carefully, and in your auth/authent logs, it's likely a files permissions, ownership, or location related.

When you configure the SSH storage server, you get to tell it which identity key to use, point it to /home/asterisk/id_pbx, and remember when entering the path to where you want to store them, use the full, not relative path.

Post Hooks Is Useful Too

Because we can run scripts or programs immediately before and after a backup is made using Hooks, we could just use Local storage on the PBX and use a Post Hook to send our backups securely to a remote server using rsync and ssh in some circumstances, a good use case would be running backups for voicemails and monitors (recorded calls), keeping one or two Local storage copies on the PBX, and have ongoing rsync updates securely sending to the backup server, uncompressed and not needing much disk space, since we are adding to, and not constantly duplicating, in such cases it's best to set Delete After (runs) a very low number, say, one to three, and run the process hourly, this keeps the Local storage lean if you have a lot of recorded calls and voicemails, without removing from the rsync storage (you should do that manually every so often), an example of a Post Hook for this is

/usr/bin/rsync -vaz -e '/usr/bin/ssh -i /home/asterisk/id_pbx'   
/var/spool/asterisk/monitor/  pbx123@backupserver:/full/path/to/backupserver/backups/pbx/pbxname/audio
Since we are using rsync over ssh, if you are concerned about ssh logins, there is rrsync (written in perl) included in later rsync packages for restricting ssh to only rsync using the earlier mentioned authorized_keys command="/some/script" key line option, where /some/script here would be the rrsync script, however this introduces other problems because it tells openssh to deny all other access to ssh including scp to whoever logs in with that key, you could work around this creating multiple users with their own keys, but messy messy messy, and again, more hassle than anyone will bother with, so I don't really recommend rrsync for our use case here, unless your sending rsyncs to a different and dedicated rsync backup server.


So hopefully now you are backing up your FreePBX system securely, this can even be applied to other asterisk based PBX's with little or no modifications, and if you're not already doing so, you should also be using a root cron job to daily and weekly backup your systems important directories, like /etc /root /home, and so on, even if you don't run a PBX, you should always backup what's important to make it easier to recover from any hardware or PEBKAP disaster.


Are Bug Reports Worth It?

I opened a bug report over FTP not using TLS, let's see how long before they close it telling me we can use FTPS if we buy commercial module XYZ at AU$150, well, that is what they told me when I reported insecure Chan_Spy :-D


Update - August 13. 2021


Well, looks like I was right, a known troll on freepbx forums decided to comment on my bug report with some crud about because we are not using sangomas distro, its tuff luck-

However it seems I have no right of reply, since Lorne Gaetz, the censorship king, has removed my reply -


Despite Mr Gaetz's actions which include closing chan_spy insecurity bug with go buy commercial module XYZ to set it, there are some at Sangoma who have taken this seriously, as the ability should now be there using "Edge's" filestore, which should make it into a future public release, so congrats to Kapil and anyone else who took this issue seriously, perhaps there is hope for FreePBX's community spirit after all.


FreePBX is a Registered Trademark of Sangoma Technologies through their purchase of Schmooze, who in turn bought the rights from the original FreePBX author, Rob Thomas. FreePBX is Open Source and completely Free Software.

The linked third party video content used in this article is copyright property of its respective owners, it is used here without permission under Australian Copyright Law Fair Dealings Exemption.


Trackbacks

No Trackbacks

Comments

Display comments as Linear | Threaded

Andre Ilhan on :

I use TrueNAS as my backups server, use scp to it, I have done this for years, but my keys are RSA and only 1024bits, I don't know a lot about generating keys other than what I read, I have just now generated my ed25519 keys as per your instructions, but I'm not sure I've done it correctly, the key length is like 1/20th the size of my RSA so I think I mucked up? They do appear to work however.

NoelB on :

Andre,
No... The ed25519 keys are much smaller, however they are far more secure, google is full of lots of explanations as to why the key is shorter.

Niko Benittitom on :

A lot of people agree with you that Sangoma is trying to drive freepbx into the ground and wreck it, they will polish their pbxacts but not the community version, those rumours are rife.

I don't know who runs Sangoma but they really don't care about us.

Jamez on :

Strange, Chris Sherwood from Crosstalk Solutions does know his stuff when it comes to FreePBX and is usually very security concious, there must be a reason why he uses FTP, perhaps he thinks FTP defaults to FTPS? I can't talk for him but he does care for his customers so must be good reason, has anybody asked him? Noel? Did you try get his reason before writing this?

NoelB on :

Jamez,
No I did not ask for his reasons, nor comment on his video, this is because he has a history of ignoring and moderating people who correct him, I've seen him do it to others over the years, people have complained about it on his forum (which I no longer for some time now visit either)

Crosstalk cannot handle criticism, even those of us who were very polite in doing so, I made 3 such posts in recent years, in relation to cabling information he gave out (since that's one of my fields of expertise), he's clearly not an expert in it but wants to come across as one, he unfortunately is one of those youtubers who wants to appear all knowing and squeaky clean, and will ignore and moderate anyone who shows otherwise, again, no matter how polite we are.

I only pointed out the wrong info because I don't want him or his followers doing things the wrong way, it benefits him too, but he doesn't see it that way, he sees it as an attack, well unless your one of his friends that is, so he will just silence you, rather than admit he was wrong he rather hide it by removing our posts and our ability to comment further.

NoelB on :

NOTICE For commenters...

I've had to deny some comments in past couple of days who more or less agreed and were in the same boat as me for crosstalk censoring our posts when correcting him, but the language used was unacceptable so I cannot approve those comments, we get the message, we are not alone in crosstalks censorship antics.

BUT folks, this article, although pointing out crosstalks fail - as he is teh one putting out this information, it is as much or more about sangomas serious failings, they are the ones truly at fault here for not using FTPS, lets keep that in mind.

thanks :-)

Ally Cat on :

yo, sangoma might be the incompetent ones but crosstalk is giving the bad advice, is he still aligned with sangoma since I thought his allegiances shifted to clearlyip which sangoma is warring with.
I don't get why crosstalk censors anyone who sets him straight on something, it's not like he has to redo his video, although he should, all he has to do is highlight the change in the videos notes and pin the comment so we see the correction, but yeah, thats make him look bad and as ya said, he don't like that

Add Comment

Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
Form options

Multiple DNSBL checks will be performed on submission of your comments.
Accepted comments will then be subject to moderation approval before displaying.