Guide to Troubleshooting the Dreaded OpenVAS 8.0 503 Status Code (service temporarily down)

April 11th at 6:41am

OpenVAS is an open source vulnerability scanner that I have used (and seen used) over the last few years. It's history goes back to 2005 as a fork of a previously open source (now commercialized) vulnerability scanner. This tool tends to be used when the dollar-cost of a commercial solution appears to outweigh the time and effort needed to maintain an effective OpenVAS installation.

The most common problem that I encounter using OpenVAS is the 503: service temporarily down error. When I see this message it almost invariably ties back to an expired self-signed certificate. I've seen this error enough times that I want to document the process in case I end up using this tool again in the future.

References

http://plugins.openvas.org/ova_503.txt
https://itsol.biz/openvas-status-code-503-status-message-service-temporarily/
https://github.com/kurobeats/OpenVas-Management-Scripts/blob/master/openvas-check-setup

(Look for the line that contains 'openvassd -s' as the command shows config details)
https://stackoverflow.com/questions/25482199/verify-a-certificate-chain-using-openssl-verify
http://openssl.6102.n7.nabble.com/Cannot-verify-self-signed-certificates-td61702.html

Background

As an Open Source tool, OpenVAS is pretty powerful. Out of the box you have the capability to setup master and worker nodes for distributed vulnerability scanning that roll-up under a single management console. Once you get used to the management interface it's not hard to configure your scan groups and dial-down the scan intensity to meet your environment's needs.

In order to maintain trust and secure communication across the networked worker nodes, OpenVAS uses a PKI system which revolves around a self-signed CA (Certificate Authority) that issues 'server' and 'client' certificates. When the certificate generation process is performed, these certificates have a short lifespan (on the order of 1 year) and when they expire can cause scans to stop working (Worker and Master nodes don't trust each other if their certificates are expired).

In the environments I've used OpenVAS, we have never leveraged the distributed scanning capabilities of the tool (largely due to time constraints and the need to just scan what we can with the time and resources we had available). Even if you don't use the distributed scanning approach, the single-node implementation utilizes the same certificate model to interact with the management console (even though they are both on the same box). This means that you will (probably, eventually) run into an issue where the certificates expire. If you scan Quarterly for compliance purposes this means that every 4th scan has a high probability of failing if you stick with the default 1 year certificate lifespan.

Symptoms

An Error message is displayed in the GSAD Web Console: Status code: 503, Status message: Service temporarily down

The following Errors are seen in the log files:

user@OpenVAS:/var/log/openvas# tail -f *
==> gsad.log <==
gsad main:  DEBUG:2017-09-20 01h31.45 utc:1143: main: gettext translation extensions are enabled (using locale "en_US.UTF-8").
gsad main:WARNING:2017-09-20 01h32.03 utc:1443: Authentication failure for 'openvasUser' from ::ffff:172.31.5.54
gsad main:WARNING:2017-09-20 01h32.06 utc:1443: Authentication failure for 'openvasUser' from ::ffff:172.31.5.54
gsad main:WARNING:2017-09-20 01h32.09 utc:1443: Authentication failure for 'openvasUser' from ::ffff:172.31.5.54
gsad main:WARNING:2017-09-20 01h32.27 utc:1443: Authentication failure for 'openvasUser' from ::ffff:172.31.5.54
gsad main:WARNING:2017-09-20 01h32.29 utc:1443: Authentication failure for 'openvasUser' from ::ffff:172.31.5.54
gsad main:WARNING:2017-09-20 01h32.31 utc:1443: Authentication failure for 'openvasUser' from ::ffff:172.31.5.54
gsad main:  DEBUG:2017-09-20 01h57.12 UTC:1443: Received Terminated signal.
gsad main:  DEBUG:2017-09-20 01h57.17 utc:2555: main: gettext translation extensions are enabled (using locale "en_US.UTF-8").
gsad main:WARNING:2017-09-20 02h07.01 UTC:2556: MHD: Error: received handshake message out of context

==> openvasmd.log <==
lib  serv:WARNING:2017-09-20 01h53.55 UTC:2418: openvas_server_verify: the certificate hasn't got a known issuer
event task:MESSAGE:2017-09-20 01h53.55 UTC:2418: Task Scan-192.168.x.x (60c51b55-638b-4b04-98df-432ba7bb7f11) could not be started by openvasUser
lib  serv:WARNING:2017-09-20 01h55.24 utc:2490: openvas_server_verify: the certificate is not trusted
lib  serv:WARNING:2017-09-20 01h55.24 utc:2490: openvas_server_verify: the certificate hasn't got a known issuer
lib  serv:WARNING:2017-09-20 01h55.30 UTC:2493: openvas_server_verify: the certificate is not trusted
lib  serv:WARNING:2017-09-20 01h55.30 UTC:2493: openvas_server_verify: the certificate hasn't got a known issuer
event task:MESSAGE:2017-09-20 01h55.30 UTC:2493: Task Scan-192.168.x.x (60c51b55-638b-4b04-98df-432ba7bb7f11) could not be started by openvasUser
lib  serv:WARNING:2017-09-20 01h57.37 UTC:2586: openvas_server_verify: the certificate is not trusted
lib  serv:WARNING:2017-09-20 01h57.37 UTC:2586: openvas_server_verify: the certificate hasn't got a known issuer
event task:MESSAGE:2017-09-20 01h57.37 UTC:2586: Task Scan-192.168.x.x (60c51b55-638b-4b04-98df-432ba7bb7f11) could not be started by openvasUser

==> openvassd.dump <==
220 NET+OS 7.4.2 FTP server ready.
Created directory: /var/lib/snmp/mib_indexes

==> openvassd.messages <==
[Wed Sep 19 10:46:01 2017][4730] Stopped scan wrap-up: Launching 1.3.6.1.4.1.25623.1.0.103962
[Wed Sep 19 10:46:01 2017][4730] Stopped scan wrap-up: Launching 1.3.6.1.4.1.25623.1.0.105780
[Wed Sep 19 10:46:01 2017][4730] Stopped scan wrap-up: Launching 1.3.6.1.4.1.25623.1.0.103964
[Wed Sep 19 10:46:01 2017][4730] Stopped scan wrap-up: Launching 1.3.6.1.4.1.25623.1.0.105781
[Wed Sep 19 10:46:01 2017][4730] Stopped scan wrap-up: Launching 1.3.6.1.4.1.25623.1.0.103963
[Wed Sep 19 10:46:01 2017][4730] Stopped scan wrap-up: Launching 1.3.6.1.4.1.25623.1.0.103240
[Wed Sep 19 10:46:02 2017][4615] Test complete
[Wed Sep 19 10:46:02 2017][4615] Total time to scan all hosts : 11123 seconds
[Fri Sep 20 01:30:49 2017][1224] Received the Terminated signal
[Fri Sep 20 01:34:19 2017][1224] openvassd 5.0.7 started

==> openvasmd.log <==
lib  serv:WARNING:2017-09-20 02h10.58 UTC:2790: openvas_server_verify: the certificate is not trusted
lib  serv:WARNING:2017-09-20 02h10.58 UTC:2790: openvas_server_verify: the certificate hasn't got a known issuer
event task:MESSAGE:2017-09-20 02h10.58 UTC:2790: Task Scan-192.168.x.x (60c51b55-638b-4b04-98df-432ba7bb7f11) could not be resumed by openvasUser

When viewing the the local bound ports on the OpenVAS node you see:

user@OpenVAS:/var/log/openvas# netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      1100/redis-server 1
tcp        0      0 0.0.0.0:9391            0.0.0.0:*               LISTEN      1224/openvassd: Wai
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1053/sshd
tcp6       0      0 :::9390                 :::*                    LISTEN      1221/openvasmd 
tcp6       0      0 :::22                   :::*                    LISTEN      1053/sshd      
tcp6       0      0 :::443                  :::*                    LISTEN      2556/gsad

Troubleshooting Process

Verify the Client certificate:

user@OpenVAS:/var/lib/openvas# openssl verify -CAfile CA/cacert.pem CA/clientcert.pem

CA/clientcert.pem: OK
Verify the Server certificate:

user@OpenVAS:/var/lib/openvas# openssl verify -CAfile CA/cacert.pem CA/servercert.pem

CA/servercert.pem: OK
Get the CACertificate to perform the next validation step:

user@OpenVAS:~# cat /var/lib/openvas/CA/cacert.pem
```
-----BEGIN CERTIFICATE-----
MIIHCDCCBPCgAwIBAgIJAN1YZP36W+R6aA1DWM...
```
Run an SQLite query on the tasks database and check if the CA Certificate in the filesystem (previous step) matches the one in the scan task. The GUID that is returned by thie sqlite3 command shows the scanner node we are looking at. In this example, there is only one scanner node). In this example you can see that the certs don't match:

user@OpenVAS:/var/lib/openvas/mgr# sqlite3 tasks.db "select uuid, ca_pub from scanners"
```
ed34c6d7-827a-4bb4-ad19-67c55cf2b6cd|-----BEGIN CERTIFICATE-----
MIIHCDCCBPCgAwIBAgIJAN1YZP36W6aCdIwGiv...
```

To Fix the Problem

Regenerate the Certificates

If your certificates don't come back as 'ok', you'll want to re-issue your client and server certificates:

systemctl stop openvas-scanner
systemctl stop openvas-manager
Generate the certificates:

openvas-mkcert -f
Create the client certificates:

openvas-mkcert-client -i -n
Use the UID returned by this command in the next step

openvasmd --get-scanners
Update the scanner and keys with the following command (Ubuntu):

openvasmd --modify-scanner "YOUR-SCANNERS-UUID-HERE (From previous step)" --scanner-ca-pub /var/lib/openvas/CA/cacert.pem --scanner-key-pub /var/lib/openvas/CA/clientcert.pem --scanner-key-priv /var/lib/openvas/private/CA/clientkey.pem

For Debian systems: openvasmd –modify-scanner “YOUR-SCANNERS-UUID-HERE (From previous step)” –scanner-ca-pub /usr/local/var/lib/openvas/CA/cacert.pem –scanner-key-pub /usr/local/var/lib/openvas/CA/clientcert.pem –scanner-key-priv /usr/local/var/lib/openvas/private/CA/clientkey.pem

Restart the Services & Rebuild

Sync up the vulnerability feeds:

openvas-nvt-sync
Restart the scanner service

systemctl start openvas-scanner
Rebuild the OpenVAS databases:

openvasmd --rebuild
Restart the OpenVAS manager service

systemctl start openvas-manager
Restart the Greenbone Security Assistant (Web UI):

systemctl start gsa

Troubleshooting Notes

I had a hard time finding where things were stored in OpenVAS. Using the 'openvassd -s' command I found these principal directories where things are stored:

- /var/lib/openvas
- /etc/openvas
- /var/log/openvas

Here's what openvassd -s looks like on Ubuntu 16.04 (for OpenVas 8):

user@OpenVAS:/var/lib/openvas/mgr# **openvassd -s**
plugins_folder = /var/lib/openvas/plugins
cache_folder = /var/cache/openvas
include_folders = /var/lib/openvas/plugins
max_hosts = 30
max_checks = 10
be_nice = no
logfile = /var/log/openvas/openvassd.messages
log_whole_attack = no
log_plugins_name_at_load = no
dumpfile = /var/log/openvas/openvassd.dump
cgi_path = /cgi-bin:/scripts
optimize_test = yes
checks_read_timeout = 5
network_scan = no
non_simult_ports = 139, 445
plugins_timeout = 320
scanner_plugins_timeout = SCANNER_NVT_TIMEOUT
safe_checks = yes
auto_enable_dependencies = yes
use_mac_addr = no
nasl_no_signature_check = yes
drop_privileges = no
unscanned_closed = yes
unscanned_closed_udp = yes
vhosts =
vhosts_ip =
report_host_details = yes
cert_file = /var/lib/openvas/CA/servercert.pem
key_file = /var/lib/openvas/private/CA/serverkey.pem
ca_file = /var/lib/openvas/CA/cacert.pem
kb_location = /var/run/redis/redis.sock
timeout_retry = 3
config_file = /etc/openvas/openvassd.conf