Unifi server controller stops working after upgrade to 5.12.35-12979-1

January 7th at 8:04am

While upgrading my home network's Unifi server installation I found that the upgrade hung for an abnormally long time and after it 'finished' the web console would not load. Investigating further it appears that somehow the port configuration for Mongo changed in this (or a prior?) version of the Unifi software which lead to it not being able to communicate with the Mongo Server. When addressing this configuration issue I found I had a disk space issue to contend with, so it's been a 'fun' morning.

References

Solution

  • Look at the log file: usr/lib/unifi/logs/server.log I see messages like this:

    [2020-01-06T23:18:39,490] <localhost-startStop-1> INFO  db     - waiting for db connection...
[2020-01-06T23:18:39,490] <localhost-startStop-1> INFO  db     - waiting for db connection...

[2020-01-07T03:35:06,249] <localhost-startStop-1> INFO  system - Hostname: UnknownHost
[2020-01-07T03:35:07,288] <localhost-startStop-1> INFO  db     - waiting for db connection...
[2020-01-07T03:35:08,793] <localhost-startStop-1> INFO  db     - Connecting to mongodb://localhost:27117
[2020-01-07T03:35:08,822] <db-server> INFO  db     - DbServer stopped
[2020-01-07T03:35:12,880] <db-server> INFO  db     - DbServer stopped
[2020-01-07T03:35:16,950] <db-server> INFO  db     - DbServer stopped

  • cd to /usr/lib/unifi

  • grep -Rni 27117

    • This is to see where the port is configured
    • I had to change the config to 27017 in a few places:
    • /usr/lib/unifi/data/system.properties
    • /usr/lib/unifi/data/unifi.init

Restarting the Unifi service, I found that it still wouldn't work. Looking in the MongoDB logs (/usr/lib/unifi/logs/mongod.log) I found these messages:

2020-01-07T03:48:46.619+0000 [initandlisten] recover : no journal files present, no recovery needed
2020-01-07T03:48:46.619+0000 [initandlisten]
2020-01-07T03:48:46.619+0000 [initandlisten] ERROR: Insufficient free space for journal files
2020-01-07T03:48:46.619+0000 [initandlisten] Please make at least 3379MB available in /usr/lib/unifi/data/db/journal or use --smallfiles
2020-01-07T03:48:46.619+0000 [initandlisten]
2020-01-07T03:48:46.619+0000 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
2020-01-07T03:48:46.619+0000 [initandlisten] dbexit:
2020-01-07T03:48:46.619+0000 [initandlisten] shutdown: going to close listening sockets...
2020-01-07T03:48:46.619+0000 [initandlisten] shutdown: going to flush diaglog...
2020-01-07T03:48:46.619+0000 [initandlisten] shutdown: going to close sockets...
2020-01-07T03:48:46.619+0000 [initandlisten] shutdown: waiting for fs preallocator...
2020-01-07T03:48:46.619+0000 [initandlisten] shutdown: lock for final commit...
2020-01-07T03:48:46.619+0000 [initandlisten] shutdown: final commit...
2020-01-07T03:48:46.619+0000 [initandlisten] shutdown: closing all files...
2020-01-07T03:48:46.619+0000 [initandlisten] closeAllFiles() finished
2020-01-07T03:48:46.619+0000 [initandlisten] journalCleanup...

Since I've been running Unifi for awhile, the MongoDB version I have is kind of old (2.6.10). I had to add a configuration to get MongoDB to start:

  • vim /etc/mongodb.conf
    • Add this to the end of the file:
    • smallfiles = true

At this point I could restart the mongoDB service, then the Unifi service and the web console came back up.