1. Peter Hansson
  2. NetBeansProxy2

Overview

HTTPS SSH

New ProxySelector for NetBeans Platform

This is an alternative / improved implementation of the standard NetBeans ProxySelector.

The code is based on NetBeans 8.0 but changes from official NetBeans 8.0 up to and including 8.2 have been forward ported to this project.

Motivation

Our application gets deployed with normal users inside corporate networks. These users - as do most corporate users - do not expect to have to specify proxy information for a single application. Most likely they don't even know what a network proxy is and even if they did they probably wouldn't know things like address and port. The user sees that other Internet-aware applications on his desktop (e.g. browser) "just works", but his NetBeans Platform based application (or IDE for that matter) cannot reach the Internet without having to fiddle with settings. That's not good.

So the goal is to make NetBeans (IDE or Platform) as good with corporate network proxies as users have gotten used to from the major browsers: IE, Firefox, Chrome, etc.

From experience most corporate networks use auto-discovery for the proxy configuration. They do not configure it on each and every workstation in the organization. This means auto-discovery is a paramount requirement.

Quick intro to centralization of network proxy configuration

If a network administrator wants to centralize proxy configuration then the known methods are:

  1. Auto-discovery mechanism. This locates where to find the configuration on the network. It is standardized through what is known as WPAD. The discovery methods used are DHCP and DNS.
  2. Selection - which proxy to use. This is done via a JavaScript snippet (called PAC script) the location of which can either be explicitly defined or defined by (1) above. The standard was originally made by Netscape who defined that 12 named helper functions must be available to the script's execution.

Even though these specs were never formally made into standards they are still very widely used and all known browsers implement them (IE, Firefox, Chrome, Safari, Opera).

Current state of affairs in NetBeans 8.x

There are a number of things that doesn't work as I would expect in NetBeans 8.x wrt Proxy selection:

PAC file evaluation with JRE8 fails

PAC file evaluation doesn't work with JRE8 (Nashorn) if the script uses the isInNet() helper function. You'll get an error saying :

    "While invoking FindProxyForURL(socket://dlc.sun.com.edgesuite.net:80, dlc.sun.com.edgesuite.net thrown javax.script.ScriptException: 
    TypeError: [RegExp /^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/] is not a function in <eval> at line number 20

No wait. Actually you'll not see this error unless you're running the IDE (or your platform app) with log level FINE or above. Which I doubt you're doing. When I saw how the JavaScript helper methods were implemented I gave up on trying to figure out why this error occurs and decided it would be better to simply rewrite that part from the ground up. There may be other scenarios that also do not work with Nashorn (i.e. use of other helper functions) but I didn't test that. Better to start over. So I did.

I've filed this issue as Bug 245116.

Missing mandatory JavaScript helper methods

Two mandatory JavaScript helper methods, dnsResolve() and myIpAddress() are not implemented in standard NetBeans. This is a known issue. This causes many PAC scripts to silently fail. Obviously this is fixed in ProxySelector2.

No auto-discovery

There's no auto-discovery of proxy configuration implemented in NetBeans on Windows. There may be two reasons for this: (1) NetBeans was never meant to auto-discover proxy configuration or (2) the implementer thought the OS would do that part for him. Judging from the code I'm leaning towards (2).

Here's how things actually works on Windows: There's a central place for Proxy Settings which you can call up by typing "Internet Options" ("Connections" tab --> "LAN Settings") in the Start Menu. The standard settings looks like this (Windows 7):

WindowsInternetSettingsDefault.png

Most people probably associate this with Internet Explorer, but it is actually a stand-alone setting. Both Chrome and Firefox can make use of this information.

NetBeans reads this information today via JNI calls to the appropriate Win API. You might think - by looking at the screenshot - that Windows will do the auto-discovery for you and store the result somewhere in the config object's memory. Unfortunately that's not how it works. "Automatically detect settings" means that the application (not Windows itself) must do some kind of automated detection. In this case NetBeans is the application.

Let me put it this way: If NetBeans is able to make use of these settings then I will expect it to arrive at the same result as say Internet Explorer. It doesn't today and that's because it doesn't implement the auto-discovery part.

What has been implemented ?

  1. Proper PAC file evaluation environment. The JavaScript helper functions have now been implemented in pure-Java and made available to the JavaScript engine. It has been tested with both JRE7 (Rhino) and JRE8 (Nashorn). The implementation is complete meaning it covers all required functions. The somewhat tricky myIpAddress() returns a sensible reply unlike in some browsers. Also the missing dnsResolve() function is implemented with a fixed timeout of 2 seconds. The ancient Netscape script, nsProxyAutoConfig.js, is now no longer needed.
  2. Auto-discovery mechanism has been implemented. This means NetBeans will now automatically locate the PAC file from the network ... just like the major browsers do. Support for discovery via DHCP has not been implemented but many browsers also do not implement this (e.g. Firefox). My guess is that 99% percent of organizations that implement auto-discovery use the DNS method, not the DHCP method. Note that while the implemented auto-discovery mechanism is cross-platform it is currently only active for Windows. Before starting to implementing in the other platform specific NetworkProxyResolvers classes one needs to understand how that platform works, meaning if the OS performs the auto-discovery for you or if we have to do it ourselves. Since I only have this knowledge on Windows then that's the only place where I've put the auto-discovery in at the moment. However I suspect that e.g. Mac OS X works much the same way. Someone else will have to verify that. This simply means that the non-Windows platform perform exactly as they do in plain vanilla NetBeans 8.x code wrt auto-discovery.
  3. PAC script result caching. Theoretically the PAC file function, FindProxyForURL(), must be executed for every URL connection the application makes. However most of the time this function will return the same result for the same input. Therefore it makes sense to cache the result from the first execution and then re-use that. The exception is if one of the time-sensitive helper functions are used (weekdayRange(), timeRange() and dateRange()) then caching will not be used.
  4. Many test cases for the JavaScript execution. Now every helper method is tested both from pure-Java environment and from inside a JavaScript snippet.
  5. If there's an error while executing the JavaScript the error is now logged at level WARNING. Previously this was FINE which meant that many errors went unnoticed.
  6. There's now support for using one's own Authenticator. See below for more information.

To see the changes from the mainstream NetBeans 8.0 code base you can do a Diff between 8dac87d and the tip. Unfortunately you cannot do this in BitBucket but you can do it from within the IDE. (Team --> Diff --> Diff to Revision --> Base to other --> press "All" filter and then choose "8dac87d" as the from-revision).

How to use this project

Just build the project and attach the resulting module as a dependency on your project. In NetBeans you can add your own ProxySelector class easily just by adding an alternative service implementation. The default one has a position value of 1000 so it is simply a matter of using a position value lower than this. This is exactly what this project does.

If you are using the IDE you can use this project as a plugin as it has been published as a NetBeans Plugin.

All in all this is project is really a drop-in replacement. There's nothing you need to do.

The module uses the same settings as the standard NetBeans ProxySelector (i.e. Tools --> Options --> General --> Proxy Settings).

WARNING: increased time before network connection is ready

The default setting in IDE, "Use System Proxy Settings", combined with the default setting in MS Windows, "Automatically detect settings", means that this new ProxySelector will have to go through the WPAD resolution process before any outgoing network connections can be made. This is the way it has to be but is unlike what happens today. Because these are the default settings this delay will affect even sites that do not use a network proxy. The NbProxySelector is created during platform initialization phase so this will potentially prolong the startup of the application/IDE.

With a well-functioning DNS the added delay will be the time for a DNS query to respond, usually <50 ms. With a non-functioning DNS setup the added delay can be at most 5 secs because the code uses a cut-off (timeout) of 5 seconds to wait for a DNS reply.

Honestly, I don't think anyone will notice this. It is just another way of saying that using some form of auto-detection (in this case WPAD) will always be slower than an explicit configuration. The WPAD resolution only happens once namely at startup.

UPDATE Oct 2015: I've implemented deferred creation of the NbProxySelector. It will now not be initialized until it is actually needed, meaning the first URLconnect operation will pay the penalty, rather than the penalty being paid during the platform startup phase. Therefore the above concerns are no longer valid.

Microsoft Windows notes

  • If system property java.net.useSystemProperties is set then NetBeans will leave the proxy selection entirely to the JDK. In other words the JDK's default ProxySelector will be used. On Windows this means that Internet Options settings will be read from Windows Registry and proxy selection will be based off that. As said - when this system property is set - the configuration will be read directly from Registry rather than using the WinHTTP API. The reason why the JDK reads from Registry rather than using the WinHTTP API is probably for legacy reasons (i.e. Windows before Windows XP and before Windows 2000). Still with this option the JDK does not perform any kind of WPAD so all-in-all it is of little use.

  • The WinHTTP API actually provides a function for doing WPAD resolution, WinHttpDetectAutoProxyConfigUrl, which could have been used in my project. This would give the benefit that it for example supports DHCP as a discovery mechanism - unlike this project. The reason why I chose not to use this native function call is that (1) I'm unfamiliar with JNA/JNI and (2) it isn't that hard to do a WPAD resolution method and by having a pure-Java solution we can reuse it on all platforms. Still, it might have been a good idea to make use of it. Windows actually has a separate service process, service name WinHttpAutoProxySvc, which takes care of the WPAD resolution. So chances are that when we make a call to WinHTTP's WinHttpDetectAutoProxyConfigUrl that it just returns a cached result from this background service, meaning that it can respond very fast. Contrast this with this project's current implementation of ProxySelector that will have to wait for the WPAD resolution process to finish before it can be used for anything.

  • The WinHTTP API also have a method for the actual proxy resolution, as in "which proxy to use for this URL?". This is the WinHttpGetProxyForUrl or its asynchronous sibling WinHttpGetProxyForUrlEx. This method does the WPAD resolution (if not done already), downloads the PAC file (if not done already) and then finally evaluates the PAC file's FindProxyForURL function to get the actual result. If we used this native Windows function then it would be Windows' JavaScript engine that would evaluate the PAC file, as opposed to using Rhino (JRE7) or Nashorn (JRE8). So why not take advantage of this native Windows method? Well, again the reason is that we would need a proper PAC file evaluation environment anyway for all non-Windows platforms.

So the conclusion so far in this project has been not to use the WinHTTP API more than is already the case. There may be an opportunity to review this decision in the future.

Why is this not part of the JDK itself ?

But it is!!

Catering for WPAD and having a correct and complete PAC file evaluation environment is actually already in the JDK.

Only problem is it sits in the deploy.jar file meaning it is accessible for Java Web Start, JavaFX, Applets and so forth .. but not for a standard Java desktop application. At least I haven't found a way to make use of the deploy.jar outside of these environments. Another problem is that deploy.jar is closed source and undocumented so it is difficult to find out exactly what is going on and what is supported and what isn't. But it goes to show that someone within Sun/Oracle already gave this some thought years ago but just didn't think there was a need for these solutions for standard Java applications.

Support for Custom Authenticator

The NB Platform installs its own Authenticator. This one is pretty useless, IMHO, as it will often just bring up a dialog which says "Authentication Required" and then "Username:" and "Password:" fields. No other information. This will confuse most users. Authentication for what ? they'll ask. Also the dialog has a tendency to become hidden behind all sorts of other dialogs in your application.

So you might just want to provide your own Authenticator. This is supported as of version 0.9.94. You can find an example in project NetBeansNetworkAuthenticator.

All you have to do is register your own implementation:

@ServiceProvider(service=java.net.Authenticator.class, position=1)
public class MyAuthenticator implements java.net.Authenticator {

The position argument is probably not needed as your implementation will be the only implementation of Authenticator in the Global Lookup.

Future of this project

The project is production ready on its own. However the goal for the project is to have the changes implemented in mainstream NetBeans. Once that hopefully happens - in one form or another - I'll kill this project. Until then you can use this project as a drop-in replacement in your own NetBeans Platform project.

License

NetBeans CDDL/GPL.

I claim no copyright on the changes I've made in this project to the core NetBeans 8.x classes.