Solr Access Control and Debugging : zetalab.de

During the migration of our web services to a new server we were upgrading the Solr search server to a more recent version. Solr is based on the Jetty application server that offers a simple integrated access control mechanism where you add IP addresses to a whitelist (in the file “server/etc/jetty.xml”) and all other addresses are denied access. Now going from Solr 5.5 to 7.3 also meant implicitly upgrading Jetty from version 9.2.13 to 9.4.8.

Trying to run Solr with our old configuration we learned that the IPAccessHandler is now deprecated, apparently due to the fact that it doesn’t (properly) cope with IPv6 addresses among other problems. Its spiritual successor is the InetAccessHandler that has a slightly different API and also doesn’t do path based access control (that we aren’t using). There’s no documentation on the use of the InetAccessHandler in the XML based Jetty configuration, though, so I had to go through trial and error.

After a while we got the access control working in a way that denied access from IP addresses that weren’t specified in the configuration. Accessing Solr/Jetty from an allowed IP turned up another problem however: the Solr web interface failed with an error 500 that corresponded to a null pointer exception in the Solr logs. Helpfully at least you get the Java class that triggered the exception, even with a line number, and a stack trace.

To find out more about the problem you can run Solr/Jetty in a Java debugger. Usually people are using Eclipse with the built-in debugger but being a commandline person I wanted to test my luck with the jdb debugger that is part of the JDK. It’s called remote debugging even if you are on the same machine because you get access to the JVM that is running the application. There are some standard JVM command line switches that you need to pass to the shell script that is starting Solr like this:

./bin/solr start -c -a "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8888"

In the case shown here the JVM debugging server binds to localhost by default. You could also provide a full qualified hostname with a port to get access from a remote server but beware that there’s no access control! People have made this work via SSH forwarding but it’s not that easy because as far as I know there are dynamically allocated ports that need to be forwarded as well.

Now that the JVM debugger server and Solr are running we can attach a debugger to it:

jdb -attach localhost:8888

To enable source level debugging I downloaded the corresponding Jetty release from Github, unpacked it and provided it as a source path to the debugger:

jdb -attach localhost:8888 -sourcepath jetty.project-jetty-9.4.8.v20171121/jetty-server/src/main/Java

As we started the debugger with the option “suspend=n” the Solr server didn’t pause but just continues to run. Now we need to place a breakpoint in the appropriate class. As we got the line number that triggered the null pointer exception we even can place it on the exact line of code. There are two different calls of the debugger “stop” command. One uses the syntax “stop in class.method” while “stop at” expects a “class:lineno”. For the class name take a look at the API docs or the corresponding Java files:

 > stop at org.eclipse.jetty.server.handler.InetAccessHandler:119

Now when a request hits the server (Solr uses port 8983 by default) it iterates over the chain of handlers until in our case it hits the breakpoint, assuming we did add the InetAccessHandler to the config. You can see the browser keeps spinning because processing can’t finish when we suspended execution with the debugger. The jdb command “locals” shows local variables, “next” runs the next statement without entering the called functions while “step” dives into the callee. Other useful commands are “print” and “dump” to inspect variables and objects. “list” shows the source code surrounding the current line of execution.

Stepping through the execution show that the HandlerWrapper (the class that InetAccessHandler inherits) has a protected variable _handler of type Handler that is null.

Actually I stopped using the debugger at this point, made an educated (by the debugger) guess, took another look at the configuration and found out that the source of the problem was a configuration error: the XML element with the id “Context” and the class “org.eclipse.jetty.server.handler.ContextHandlerCollection” needs to be wrapped in a “handler” element so the “_handler” actually gets allocated and is no longer null.

You can surely get to that point through jdb although it will be hard because the Jetty source code is a maze of directories and design patterns you will to need understand at least partially. You need to be familiar with the Java Reflection API as well because Jetty uses it to build its runtime structures from the XML configuration. Take a look at “java/org/eclipse/jetty/xml/XmlConfiguration.java” as an entrypoint.

So there you have it. That’s how the config (“server/etc/jetty.xml”) is supposed to look when you want to use the InetAccessHandler class:

       <Set name="handler">
         <New id="Handlers" class="org.eclipse.jetty.server.handler.HandlerCollection">
           <Set name="handlers">
             <Array type="org.eclipse.jetty.server.Handler">
                 <Item>
                    <New class="org.eclipse.jetty.server.handler.InetAccessHandler">
                      <Call name="include">
                        <Arg>127.0.0.1</Arg>
                      </Call>
                      <Call name="include">
                        <Arg>1.2.3.4</Arg>
                      </Call>
                      <Call name="include">
                        <Arg>2.3.4.5</Arg>
                      </Call>
                      <Set name="handler">
                        <New id="Contexts" class="org.eclipse.jetty.server.handler.ContextHandlerCollection"/>
                      </Set>
                    </New>
               </Item>

I actually think it’s poor design that Jetty doesn’t detect misconfigurations like this. I don’t fully understand the configuration syntax so I don’t know if it makes any sense to have the “ContextHandlerCollection” without being wrapped inside a Handler set but obviously you can create configurations that lead to undefined behaviour and null pointer exceptions without being warned.