Getting around Dell iDRAC Java SSLHandshakeException

Sep 8, 2021 · 5 min read · Dell iDRAC Java error ·

Share on:

TL;DR

The steps below assume network connectivity to iDRAC has been established. Follow the steps below to get past this error and launch the iDRAC console. Read on for the analysis that lead to these steps.

Download avctKVM.jar from https://<your iDRAC IP/hostname here>/software/avctKVM.jar
Edit the java.security policy file and comment out the lines referring to jdk.tls.disabledAlgorithms. On my Linux machine the file is at the following path /etc/java-11-openjdk/security/java.security.
Run the following command to start the console viewer application java -cp avctKVM.jar com.avocent.idrac.kvm.Main ip=<iDRAC IP/hostname here> kmport=5900 vport=5900 user=<iDRAC user> passwd=<iDRAC password>

Note: Performing step 2 above will leave your machine vulnerable to Java applications making connections using insecure encryption algorithms and protocols. Make sure to revert changes after your activities are complete.

What is iDRAC?

I have a Dell server in my homelab and I needed a way to control it without a monitor or keyboard attached. Luckily it has iDRAC. “The Integrated Dell Remote Access Controller (iDRAC) is an out-of-band management platform on certain Dell servers.” At the risk of several @’s I’ll compare it to IPMI on Supermicro boards: it gives an administrator access to a console on the machine over a network connection so that maintenance activities can take place.

With a quick network connection in place to the iDRAC interface on the motherboard, I had access to iDRAC and could control my server that was powered off.

iDRAC Console Access

Afer logging in - default credentials for Dell iDRAC were root/calvin in my server’s case - I powered on the server and tried to launch the Virtual Console. I was prompted to download a viewer.jnlp file. Without going too deep down that rabbit hole - JNLP stands for Java Network Launch Protocol - I needed to find a way to run this file on my Linux machine. THe file is XML and so can be inspected using a text editor.

First there was a section listing what looked like a Java class and arguments to pass:

 1<application-desc main-class="com.avocent.idrac.kvm.Main">
 2   <argument>ip=iDRAC-hostname.domain</argument>
 3   <argument>vm=1</argument>
 4   <argument>helpurl=https://iDRAC-hostname.domain:443/help/contents.html</argument>
 5   <argument>title=iDRAC-hostname%2C+PowerEdge+T420%2C++slot+%2C+User%3A+root</argument>
 6   <argument>user=[digits]</argument>
 7   <argument>passwd=[digits]</argument>
 8   <argument>kmport=5900</argument>
 9   <argument>vport=5900</argument>
10   <argument>apcp=1</argument>
11   <argument>chat=1</argument>
12   <argument>F1=1</argument>
13   <argument>custom=0</argument>
14   <argument>scaling=15</argument>
15   <argument>minwinheight=100</argument>
16   <argument>minwinwidth=100</argument>
17   <argument>videoborder=0</argument>
18   <argument>version=2</argument>
19 </application-desc>

So the class I was after is called com.avocent.idrac.kvm.Main. Cool, but where was this class defined? Several <resources> elements follow the previous section and point Java archive files for different OS architectures:

 1<resources>
 2   <j2se version="1.6+"/>
 3   <jar href="https://iDRAC-hostname.domain:443/software/avctKVM.jar" download="eager" main="true" />
 4 </resources>
 5 <resources os="Windows" arch="x86">
 6   <nativelib href="https://iDRAC-hostname.domain:443/software/avctKVMIOWin32.jar" download="eager"/>
 7   <nativelib href="https://iDRAC-hostname.domain:443/software/avctVMAPI_DLLWin32.jar" download="eager"/>
 8 </resources>
 9 <resources os="Windows" arch="amd64">
10   <nativelib href="https://iDRAC-hostname.domain:443/software/avctKVMIOWin64.jar" download="eager"/>
11   <nativelib href="https://iDRAC-hostname.domain:443/software/avctVMAPI_DLLWin64.jar" download="eager"/>
12 </resources>
13 <resources os="Windows" arch="x86_64">
14   <nativelib href="https://iDRAC-hostname.domain:443/software/avctKVMIOWin64.jar" download="eager"/>
15   <nativelib href="https://iDRAC-hostname.domain:443/software/avctVMAPI_DLLWin64.jar" download="eager"/>
16 </resources>

So the class com.avocent.idrac.kvm.Main is defined in the avctKVM.jar. I downloaded the file from the first link, https://iDRAC-hostname.domain:443/software/avctKVM.jar and put together a command to run it based on the arguments observed previously. The command ended up being the following:

1java -cp avctKVM.jar com.avocent.idrac.kvm.Main ip='<iDRAC IP address here>' kmport=5900 vport=5900 user='<most likely "root">' passwd='<default for user "root" is calvin>'

Breaking down the command:

java -cp avctKVM.jar com.avocent.idrac.kvm.Main - call the class from the specified JAR file
ip - IP or FQDN of iDRAC device
kmport=5900 vport=5900 - I assumed it needed to know which ports to connect to for keyboard and mouse (kmport) and image (vport) so I included these
user= passwd= - the same credentials used to login to the iDRAC interface in the first place (defaults are root/calvin)

javax.net.ssl.SSLHandshakeException: No appropriate protocol

I got the following error after running that command:

Not very helpful, but the terminal showed the underlying issue:

 1javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
 2	at java.base/sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:170)
 3	at java.base/sun.security.ssl.ClientHandshakeContext.<init>(ClientHandshakeContext.java:98)
 4	at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:221)
 5	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:433)
 6	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
 7	at com.avocent.d.a.a.a(Unknown Source)
 8	at com.avocent.d.a.a.a(Unknown Source)
 9	at com.avocent.d.a.a.a(Unknown Source)
10	at com.avocent.d.c.b.a(Unknown Source)
11	at com.avocent.a.a.t.g(Unknown Source)
12	at com.avocent.a.a.t.a(Unknown Source)
13	at com.avocent.app.c.j.m(Unknown Source)
14	at com.avocent.app.c.j.d(Unknown Source)
15	at com.avocent.idrac.kvm.a.d(Unknown Source)
16	at com.avocent.idrac.kvm.Main.a(Unknown Source)
17	at com.avocent.idrac.kvm.Main.main(Unknown Source)

The application is using an unsupported protocol. Makes sense, I haven’t updated iDRAC to a modern version that supports modern TLS/SSL settings. Java is looking out for the integrity and confidentiality of the connection by not downgrading it to use weak encryption. This is a good thing, you don’t want weak encryption when you’re passing credentials and accessing servers out of band over insecure networks. You also don’t want to expose iDRAC to insecure networks but that’s another story.

Since I was accessing it over my local network that is fairly trustworthy I needed a bypass. Turns out it was straight forward and only required editing the Java policy file at /etc/<Java runtime>/security/java.security. On my machine the path is /etc/java-11-openjdk/security/java.security. The settings that need to be changed in this file are the values in jdk.tls.disabledAlgorithms.

Several algorithms with known issues are disabled by default including SSLv3, TLSv1 and TLSv1.1, and RC4. In the original file the settings were the following:

1jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
2    DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, \
3    include jdk.disabled.namedCurve

I commented out the above section and ran the command again. Success!

Note that this applies across all Java applications running on the system therefore they could all be downgraded to using SSLv3 after this chanage is made. According to Oracle documentation there is a way of specifying Java policy at run time which would be a more secure way of approaching this problem.