The other day I was helping a friend troubleshoot an issue with his Ubiquiti
Cloud Key. At issue was the Unifi Controller itself and in his specific case it just happened to be running on a Cloud Key.
[The problem]
When attempting to generate and download a controller backup, the following error is displayed: There was an error generating a new backup file. Backup failed.
Furthermore, the cloudkey controller seemed to have stopped generating automatic backup files (called autobackups) and would freeze if an end user attempted to “compact” the database from within the Unifi UI.
[The solution]
After some research, it was determined that the mongo database on the cloud key had been damaged somehow and needed to be repaired. This repair procedure resolves the issue: https://help.ubnt.com/hc/en-us/articles/360006634094-UniFi-Network-Controller-Repairing-Database-Issues-on-the-UniFi-Controller#2
[Further explanation]
While researching this issue, I found that the Unifi Cloud Key is essentially a POE powered custom Linux appliance with a micro-sd memory card for storage. The appliance runs the Unifi Controller application which can then be synced with a users existing Ubiquiti Unifi Cloud account allowing for remote management of all devices the Controller manages.
Troubleshooting for me began with getting logged into the Cloud Key via ssh. All data related to the Unifi Controller appear to be stored in /usr/lib/unifi/. The most useful item in pointing out what was going wrong with the Cloud Key was the /usr/lib/unifi/logs/server.log file. In it several instances of the following error were found:
Write failed with error code 13636 and error message ‘file /usr/lib/unifi/data/db/ace_stat.3
Googling that specific error led to the database repair article referenced above which ended up fixing the issue.
Here is the full output of all the commands with commands bolded:
root@a2cloudkey:/tmp# service unifi stop
root@a2cloudkey:/tmp# mongod –dbpath /usr/lib/unifi/data/db –smallfiles –logpath /usr/lib/unifi/logs/server.log –repair
Thu Feb 7 15:29:28.695
Thu Feb 7 15:29:28.696 warning: 32-bit servers don’t have journaling enabled by default. Please use –journal if you want durability.
Thu Feb 7 15:29:28.696
all output going to: /usr/lib/unifi/logs/server.log
log file [/usr/lib/unifi/logs/server.log] exists; copied to temporary file [/usr/lib/unifi/logs/server.log.2019-02-07T23-29-28]
root@a2cloudkey:/tmp# mongod –dbpath /usr/lib/unifi/data/db –smallfiles –logpath /usr/lib/unifi/logs/server.log –journal –fork
about to fork child process, waiting until server is ready for connections.
forked process: 13573
all output going to: /usr/lib/unifi/logs/server.log
log file [/usr/lib/unifi/logs/server.log] exists; copied to temporary file [/usr/lib/unifi/logs/server.log.2019-02-07T23-36-28]
child process started successfully, parent exiting
root@a2cloudkey:/tmp# mongo < /tmp/CK_repair.js
MongoDB shell version: 2.4.10
connecting to: test
pruning data older than 7 days (1548977811201)…
switched to db ace
pruning 668 entries (total 726) from alarm…
pruning 9819 entries (total 25034) from event…
pruning 0 entries (total 0) from guest…
pruning 94633 entries (total 103072) from rogue…
pruning 256 entries (total 312) from user…
pruning 0 entries (total 0) from voucher…
{ “ok” : 1 }
{ “ok” : 1 }
switched to db ace_stat
pruning 658 entries (total 699) from stat_5minutes…
pruning 25401 entries (total 25495) from stat_archive…
pruning 418 entries (total 418) from stat_daily…
pruning 4668 entries (total 8202) from stat_dpi…
pruning 5103 entries (total 5103) from stat_hourly…
pruning 245 entries (total 287) from stat_life…
pruning 0 entries (total 0) from stat_minute…
pruning 45 entries (total 45) from stat_monthly…
{ “ok” : 1 }
{ “ok” : 1 }
bye
root@a2cloudkey:/tmp# mongod –dbpath /usr/lib/unifi/data/db –smallfiles –logpath /usr/lib/unifi/logs/server.log –shutdown
Thu Feb 7 15:39:49.441
Thu Feb 7 15:39:49.442 warning: 32-bit servers don’t have journaling enabled by default. Please use –journal if you want durability.
Thu Feb 7 15:39:49.443
killing process with pid: 13573
root@a2cloudkey:/tmp# service unifi start