What to do when a module needs a native library?

[t***@sun.com]

Hi,

My module uses a native library rxtxSerial.dll, so I put the rxtxSerial.dll in nb4.1/module/lib. (nb4.1/module is where the module jar file lives). I also included the rxtxSerial.dll in the module's build.xml's files-init target. However, I got a runtime exception.

One thing to note is, before I moved my module to be part of the netbeans ide build ( i.e when the module was installed as a plug-in through the update center) I would put the rxtxSerial.dll in the user directory's module/lib, and it worked fine. I read in one of the documents that the module/lib in both the user and installation directory will be added to the search path for System.loadlibrary() from the module's code. But the document also says this is an unsupport way. Is this still valid? What should I do so that my module can use this library?

Thanks.

Tim Shi

[Jesse Glick]

Post by t***@sun.com
My module uses a native library rxtxSerial.dll, so I put the
rxtxSerial.dll in nb4.1/module/lib. (nb4.1/module is where the module
jar file lives). I also included the rxtxSerial.dll in the module's
build.xml's files-init target. However, I got a runtime exception.

Don't know, should work fine. Check your file structure again. The
extbrowser module does the same thing in the NB IDE and it works.

Post by t***@sun.com
One thing to note is, before I moved my module to be part of the
netbeans ide build ( i.e when the module was installed as a plug-in
through the update center) I would put the rxtxSerial.dll in the user
directory's module/lib, and it worked fine.

OK...

Post by t***@sun.com
I read in one of the
documents that the module/lib in both the user and installation
directory will be added to the search path for System.loadlibrary()
from the module's code.

Yes.

Post by t***@sun.com
But the document also says this is an unsupport[ed] way.

Ignore that.

-J.

[Jesse Glick]

[...] I read in one of the documents that the module/lib in both the
user and installation directory will be added to the search path for
System.loadlibrary() from the module's code.

"modules/lib" not "module/lib"

The more precise statement is that for a module in $dir/$file.jar, the
search path will contain $dir/lib, so System.loadLibrary("foo") will
find $dir/lib/libfoo.so (Unix) or $dir/lib/foo.dll (Windows). The
current document is misleading in stating that you can use either the
"user" or "installation" directory. Wherever the module is, that is
where the native lib must go.

-J.

[akochnev]

I'm responding to this old thread as it's referenced from a few places in
google searches .

I was trying to wrap the MsSQL JDBC driver and make sure that integrated
authentication works (which requires a bundled dll). When I placed the dll
in release/modules/lib the code from the mssql driver was unable to find the
dll. Moving the dll into release/modules/ext/lib did the trick.

The FAQ entry at http://wiki.netbeans.org/DevFaqNativeLibraries has been
updated with the following :



How do I add native library to a Library wrapper module
------------------------------------------

There is a little twist to wrapping a third party library that uses a dll
(e.g. like the MsSQL JDBC Driver) - the .dll needs to be placed in a lib
subdirectory relative to the third party jar (release/modules/ext/lib/ ) so
that when the third party library tries to use it the dll is found.

The API Reference: JNI in modules API reference speaks about how the .dll
needs to be placed in the lib directory relative to the module jar file.
That all works if code in the module itself uses the native library;
however, when it's the third party's code in release/modules/ext uses it, if
you place your .dll inside of release/modules/lib it's unable to find the
.dll .

[Jesse Glick]

Post by akochnev
I was trying to wrap the MsSQL JDBC driver and make sure that integrated
authentication works (which requires a bundled dll). When I placed the dll
in release/modules/lib the code from the mssql driver was unable to find the
dll. Moving the dll into release/modules/ext/lib did the trick.

I'm pretty sure that's wrong. StandardModule.OneModuleClassLoader in 6.9+ (*) specifically looks in <cluster>/modules/lib/ (plus optional architecture & OS subdirs). It
should not matter whether the code calling System.loadLibrary resides in the main module JAR or a Class-Path extension, since all are defined by the same class loader.

(*) In 6.8- it looks in lib/ starting from the parent directory of the module JAR. For the purposes of this discussion the effect should be the same.

[Aleksandar Kochnev]

Jesse,
When you say it's wrong, do you mean that I should file a bug for this behavior (considering that I've double and triple confirmed the behavior, at least when trying to wrap the mssql driver) ? Or do you mean that the instruction to put the dll under ext/lib is incorrect but somehow manages to produce the expected behavior when working w/ the Microsoft SQL Server driver?

Alex Kochnev
Architect, Quality Assurance
CommerceHub
255 Fuller Road Suite 327
Albany, NY 12203
518.810.0700 x3876
http://www.commercehub.com

-----Original Message-----
From: Jesse Glick [mailto:***@oracle.com]
Sent: Sunday, November 28, 2010 8:49 AM
To: ***@netbeans.org
Subject: [nbdev] Re: What to do when a module needs a native library?

Post by akochnev
I was trying to wrap the MsSQL JDBC driver and make sure that
integrated authentication works (which requires a bundled dll). When I
placed the dll in release/modules/lib the code from the mssql driver
was unable to find the dll. Moving the dll into release/modules/ext/lib did the trick.

I'm pretty sure that's wrong. StandardModule.OneModuleClassLoader in 6.9+ (*) specifically looks in <cluster>/modules/lib/ (plus optional architecture & OS subdirs). It should not matter whether the code calling System.loadLibrary resides in the main module JAR or a Class-Path extension, since all are defined by the same class loader.

(*) In 6.8- it looks in lib/ starting from the parent directory of the module JAR. For the purposes of this discussion the effect should be the same

[Jesse Glick]

Post by Aleksandar Kochnev
do you mean that I should file a bug for this behavior

Yes; it should work to place DLLs under modules/lib.

Post by Aleksandar Kochnev
do you mean that the instruction to put the dll under ext/lib is incorrect but somehow manages to produce the expected behavior

Also yes; it should _not_ work to place DLLs under modules/ext/lib.

[akochnev]

Jesse,

I followed this a bit further and discovered an issue w/ my setup which was probably contributing to the issue ( I had manually copied the .dll into the netbeans/bin directory).

Prior to removing the .dll from the netbeans/bin directory, I was consistently seeing the behavior I described in the previous posting (the mssql server jdbc connection succeeding when I placed a dll in release/modules/ext/lib, and not working when placed in release/modules/lib). Fortunately for the bug that I was going to file, and unfortunately for what I was trying to accomplish, after I removed the .dll from the netbeans/bin directory, the JDBC driver setup completely stopped being able to load the dll.

This clears up the potential confusion as to why it worked in ext/lib. However, the problem is that the mssql server driver jar still reports not being able to find the dll when I put it in modules/lib. The MSSQL documentation clearly states that the dll should either be in the path (e.g. windows\system32) or adding the path to the dll folder to java.library.path . However, it was my impression that adding the dll to modules/lib was supposed to address that.

Any ideas of what I could do next to figure this out ? I tried a few different permutations (e.g. adding the dll to OS and architecture specific folders under modules/lib) but with no positive results.

[Jesse Glick]

Post by akochnev
Prior to removing the .dll from the netbeans/bin directory, I was consistently seeing the behavior I described in the previous posting (the mssql server jdbc
connection succeeding when I placed a dll in release/modules/ext/lib, and not working when placed in release/modules/lib).

Maybe because when in release/modules/ext/lib, it had no effect at all, and the version from netbeans/bin was loaded. Whereas when in release/modules/lib, there was some
kind of linker conflict.

Post by akochnev
the problem is that the mssql server driver jar still reports not being able to find
the dll when I put it in modules/lib. The MSSQL documentation clearly states that the dll should either be in the path (e.g. windows\system32) or adding the path to
the dll folder to java.library.path . However, it was my impression that adding the dll to modules/lib was supposed to address that.

It is supposed to, yes. Are you sure the driver JAR is being loaded from the module class loader, i.e. it is a Class-Path extension of the module? What does the driver
call to load the DLL? If you debug org.netbeans.StandardModule.OneModuleClassLoader.findLibrary what happens?

[akochnev]

Post by Jesse Glick
Are you sure the driver JAR is being loaded from the module class loader, i.e. it is a Class-Path extension of the module? What does the driver
call to load the DLL? If you debug org.netbeans.StandardModule.OneModuleClassLoader.findLibrary what happens?

In the project, I have the jar wrapped up as a library wrapper module and the jar is listed in the project.xml as follows:
<class-path-extension>
<runtime-relative-path>ext/sqljdbc4.jar</runtime-relative-path>
<binary-origin>release/modules/ext/sqljdbc4.jar</binary-origin>
</class-path-extension>

Now, about the jar being loaded through the module classloader - it seems to me that it is not being loaded through that, as when I try to connect using the said driver, I get the stacktrace below ( and the StandardModule.OneModuleClassLoader is not in the it) .

What's interesting is that when I add a System.loadLibrary("sqljdbc_auth") from my module installer, the dll is found as expected when the DLL is inside of modules/lib. However, when it's called through the driver, it evidently doesn't find it.

WARNING [com.microsoft.sqlserver.jdbc.internals.AuthenticationJNI]: Failed to load the sqljdbc_auth.dll cause :- no sqljdbc_auth in java.library.path
INFO [org.netbeans.modules.db.explorer.action.ConnectUsingDriverAction]
java.lang.UnsatisfiedLinkError: no sqljdbc_auth in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1734)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1028)
at com.microsoft.sqlserver.jdbc.AuthenticationJNI.<clinit>(AuthenticationJNI.java:32)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:1902)
Caused: com.microsoft.sqlserver.jdbc.SQLServerException: This driver is not configured for integrated authentication.
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1352)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:2329)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:1905)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$000(SQLServerConnection.java:41)
at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:1893)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4575)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1400)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1045)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:817)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:700)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:842)
at org.netbeans.modules.db.explorer.DbDriverManager.getConnection(DbDriverManager.java:131)
at org.netbeans.modules.db.explorer.DatabaseConnection.doConnect(DatabaseConnection.java:752)
at org.netbeans.modules.db.explorer.DatabaseConnection.access$100(DatabaseConnection.java:106)
[catch] at org.netbeans.modules.db.explorer.DatabaseConnection$2.run(DatabaseConnection.java:808)
at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1418)
at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1957)

[Jesse Glick]

it seems to me that [the JAR] is not being loaded through [the module class loader], as when I try to connect using the said driver, I get the stacktrace below ( and
the StandardModule.OneModuleClassLoader is not in the it) .

I wouldn't expect any stack traces to mention the defining class loader. System.loadLibrary asks the loader to find the library, then continues to try to load it. Just
verify that AuthenticationJNI.class.getClassLoader() == YourInstaller.class.getClassLoader().

What's interesting is that when I add a System.loadLibrary("sqljdbc_auth") from my module installer, the dll is found as expected when the DLL is inside of
modules/lib.

A possible workaround, since IIRC System.loadLibrary is idempotent (i.e. Runtime.loadLibrary specifies "If this method is called more than once with the same library
name, the second and subsequent calls are ignored.").

However, when it's called through the driver, it evidently doesn't find it.

No further tips, try a debugger.

[Roger Martin]

Post by akochnev
What's interesting is that when I add a System.loadLibrary("sqljdbc_auth")
from my module installer, the dll is found as expected when the DLL is
inside of modules/lib. However, when it's called through the driver, it
evidently doesn't find it.

For development debugging try System.load("<full local path to
file>/sqljdbc_auth.dll");
instead. This can reveal it is finding the sqljdbc_auth.dll but
failing on its further dll dependencies. These need to be findable
too in the lib or on the library path