Hyper-V, ICS, RRAS NAT and port-forwarding on Windows Server 2012

With some friends we bought a quite strong computer and set up Window Server 2012 with Hyper-V virtualized machines for our projects. Computing power is much cheaper this way than in notebooks, so we can use lighter notebooks to be generally used as portable remote terminals. It works out quite well. RDP can be used well over 3G too.

However we started to face network problems on the server. At first, the computer was located on a company site, so the network configuration looked like this:

This required no special configuration, everything just worked as it should. The host machine and the VMs got an internal IP address from the router. To access VMs from the internet, we added port forwarding on the router.

Internal Switch

Then we decided to put the host machine to a server hosting room with a dedicated IP address. That required us to change the network configuration, as the External Switch would no longer work without a router in front of it.

With the Internal Switch the virtual machines share a virtual network that can also be accessible from the host machine. To have internet on the VMs, you have to set up NAT.

Internet Connection Sharing

Our very first try of configuration

1. The primary NIC has a fix IP address with fix gateway and DNS.

2. The NIC has Internet Connection Sharing turned on:

If the "Home networking connection" combobox doesn't appear, don't worry. If there's only one other elegible connection, then it's unshown.

When you click OK, the Internal Switch should be configured to 192.168.137.1/255.255.255.0

Havin this done, and having the internal switch assigned to the VMs, the VMs should now properly have internet connection without any further configuration. They get IP addresses like 192.168.137.x and 192.168.137.1 as gateway. If they don't, try disabling and enabling the internal switch, or configure them (while running) to have no network (Not connected), then configure them to use Internal Switch again. These might help.

If you want to open an RDP session to a VM, click Settings on the ICS screen above and add a rule:

Also make sure to enable inbound connections on the port 3390 of the host machine. Now you can open an RDP session on hostmachine:3390 and it will be forwarded to the VM. Of course, also make sure to enable remote access on the VM.

When testing this, make sure to open the connection from the outside internet. As I've found it in a random blog post, ICS doesn't route port forwarding from the intranet, so if you try hostmachine:3390 from the hostmachine itself, it probably won't work, but if you try from 'outside', it will.

This solution can work, but has some drawbacks:

• The settings above sometimes just does not work. I haven't yet figured out what to restart and in what order, but sometimes you just have to reconfigure the entire thing. Enablind and disabling the Internal Switch, detaching and reattaching it to the VM, disabling and enabling ICS are options you can perform in any order.
• Altought network connection over NAT is just as fast as the host connection, RDP over the port forwarding is unusably slow. I've found some forum posts about how to make it faster (NIC interface configurations) but they didn't work for me.
• In some constellation, if I connect to a VM, it asks me for my credentials, but then the screen remains blank. Also my concurrent RDP connection to the host hangs up. After some 20 seconds the VM RDP connection closes and the host connection resumes. If this happens, try reconfiguring everything.
• After rebooting the system I usually ended up losing RPD connectivity to the host machine. Every other thing worked: RDP to the VMs and even http worked on the host machine (apache server). To prevent this I had to explicitly add a port forwarding from hostmachine:3389 to 127.0.0.1:3389. Weird.

Also, at my first try the network connection to have packets lost. If I enabled ICS, the host RDP session only worked for 10 seconds, then hanged for another 10, then it took 10 seconds to reconnect. This was solved by updating my NIC driver with a newer one from the vendor (Intel).

Usually you should be able to connect to the VMs from the host by connecting to eg. 192.168.137.2:3389. They won't have a ping though, so don't worry about that.

Routing and Remote Access

The same configuration can be acheived with RRAS on windows servers. This feature cannot be used together with ICS. Once you added this role in the server manager, open "mmc" and add the routing and remote access snap-in and add the local server.

You can configure NAT by either selecting NAT directly in the wizard or by selecting custom configuration and just adding bare NAT, and then later adding interfaces yourself.

To use NAT, you have to have two interfaces added:

• the external NIC added as "public interface connected to the internet"
• the internal switch added as "private interface"

There are IPv4 settings if you right/click the local server. My setting is DHCP here, so no address pool is configured. In the NAT properties, Address Assignment tab:

192.168.0.0/255.255.255.0 is configured. The wizard will configure this for you if you previously configure your Internal Switch:

This produces the same stuff that ICS has. To add port forwarding you can go to the properties window of the public interface and click the services and ports tab. However, this has the same drawbacks as ICS port forwarding had: slow RDP sessions. They must have really messed up something there.

With this configuration you also should have access to VMs from the host machine directly, using their 192.168.0.x addresses.

I started to look for third-party port-forwarding tools, as it's quite an easy to implement stuff. Rinetd worked fine, but it doesn't run as a windows service. After a bit of googling I found that windows itself can do portforwarding (of course): http://technet.microsoft.com/en-us/library/cc731068(v=ws.10).aspx . So you can do this from an administrator command line:

netsh interface portproxy add v4tov4 3397 192.168.0.2 3389

Use netsh interface portproxy to display additional options. With this configuration, port redirection works fine and RDP sessions work properly. This command also implicitly creates a firewall exception for the public port.

In case stuff stop working, reconfigure NAT from scratch. Also some rebooting will help.

Summary

Having all this work properly took me painful weeks. Other forum posts also state that this functionality of the windows server is quite unstable. I feel lucky that I could finally come to a point where it works.

While this was not working I used VirtualBox and VMWare virtualization. They have this port-forwarding and NAT feature out-of-the-box and it just works with a few clicks. It's a shame that Hyper-V delegates this to ICS or RRAS, and finally you have to configure complicated stuff that don't finally work by default. Also, tutorials on technet just don't go into important details. They just say, 'enable NAT' and it will work. But it won't.

Also note that with vmware and virtualbox, if the host machine was connected to the company VPN, the virtual machines also accessed company intranet resources. With RRAS and ICS, VMs have to connect to VPN individually. Maybe there are routing settings that can solve this more easily, but after this torture, I wouldn't start configuring it.

I hope this helps other folks out there :)

JBoss AS 7 and PostgreSQL

So, I'm having some fun migrating a major project to EE6. I've been playing around with Glassfish 3.1.2, but now I'm evaluating JBoss AS 7.1.1.

First of all, I tried to use hibernate 4.1.1 bundled with my WAR file, but this caused this exception:

Caused by: java.lang.ClassCastException: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider cannot be cast to org.hibernate.service.jdbc.connections.spi.ConnectionProvider
 at org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator.instantiateExplicitConnectionProvider(ConnectionProviderInitiator.java:187)

I haven't yet found a way to resolve this, so for then I just went on using hibernate 4.0.1 in the AS. I guess it's mainly because the persistence unit in my application is loaded with the 4.1.1 jars, but the persistence unit, as it is deployed in the AS (not inside the WAR), is loaded by the 4.0.1 version. I'll give updates on this.

So when staying with 4.0.1, everything deployed fine, but when performing an operation, I got this:

Caused by: org.postgresql.util.PSQLException: Large Objects may not be used in auto-commit mode.
 at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:200)
 at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:172) 

Althought there's a hibernate.connection.autocommit setting that can be used in the persistence.xml, the problem was solved by enabling "Use JTA?" in the Datasource settings on the JBoss admin console.

A JVM networking bug

My former colleagues sent me a strange error from a production jboss instance running on windows server 2003. Occasionally an AV terminates the VM:


EXCEPTION_ACCESS_VIOLATION (0xc0000005)


C  [ntdll.dll+0x2b583]  wcscpy+0x108
C  [ntdll.dll+0x2ba81]  RtlTimeFieldsToTime+0x2cb
C  [ntdll.dll+0x2b646]  wcscpy+0x1cb
C  [msvcr71.dll+0x218a]  free+0x39
C  [net.dll+0x70fd]  Java_java_net_SocketInputStream_socketRead0+0x1c6
J  java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I


From another VM:


C  [ntdll.dll+0x2be3e]
C  [ntdll.dll+0x2b561]
C  [ntdll.dll+0x2ba81]
C  [ntdll.dll+0x2b646]
C  [msvcr71.dll+0x218a]
C  [net.dll+0x7129]
j  java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I+0
j  java.net.SocketInputStream.read([BII)I+84
j  org.apache.coyote.http11.InternalInputBuffer.fill()Z+59


Checking some forums and stuff didn't help too much. There's even a bug for this http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5040096 (I tried to comment on that bug, but this sun portal is not a friend of mine lately ... ) (also https://forums.oracle.com/forums/thread.jspa?threadID=1582665 ).
What I could figure out that the error must be in the JVM net.dll. It's quite strange though that the error seems to come only on "Windows Server 2003 family Build 3790 Service Pack 2".


I checked the source of the socketRead0 method:

/*
 * Class:     java_net_SocketInputStream
 * Method:    socketRead
 * Signature: (Ljava/io/FileDescriptor;[BIII)I
 */
JNIEXPORT jint JNICALL
Java_java_net_SocketInputStream_socketRead0(JNIEnv *env, jobject this,
                                            jobject fdObj, jbyteArray data,
                                            jint off, jint len, jint timeout)
{
    char *bufP;
    char BUF[MAX_BUFFER_LEN];
    jint fd, newfd;
    jint nread;

    if (IS_NULL(fdObj)) {
        JNU_ThrowByName(env, JNU_JAVANETPKG "SocketException", "socket closed");
        return -1;
    }
    fd = (*env)->GetIntField(env, fdObj, IO_fd_fdID);
    if (fd == -1) {
        NET_ThrowSocketException(env, "Socket closed");
        return -1;
    }

    /*
     * If the caller buffer is large than our stack buffer then we allocate
     * from the heap (up to a limit). If memory is exhausted we always use
     * the stack buffer.
     */
    if (len <= MAX_BUFFER_LEN) {
        bufP = BUF;
    } else {
        if (len > MAX_HEAP_BUFFER_LEN) {
            len = MAX_HEAP_BUFFER_LEN;
        }
        bufP = (char *)malloc((size_t)len);
        if (bufP == NULL) {
            /* allocation failed so use stack buffer */
            bufP = BUF;
            len = MAX_BUFFER_LEN;
        }
    }


    if (timeout) {
        if (timeout <= 5000 || !isRcvTimeoutSupported) {
            int ret = NET_Timeout (fd, timeout);

            if (ret <= 0) {
                if (ret == 0) {
                    JNU_ThrowByName(env, JNU_JAVANETPKG "SocketTimeoutException",
                                    "Read timed out");
                } else if (ret == JVM_IO_ERR) {
                    JNU_ThrowByName(env, JNU_JAVANETPKG "SocketException", "socket closed");
                } else if (ret == JVM_IO_INTR) {
                    JNU_ThrowByName(env, JNU_JAVAIOPKG "InterruptedIOException",
                                    "Operation interrupted");
                }
                if (bufP != BUF) {
                    free(bufP);
                }
                return -1;
            }

            /*check if the socket has been closed while we were in timeout*/
            newfd = (*env)->GetIntField(env, fdObj, IO_fd_fdID);
            if (newfd == -1) {
                NET_ThrowSocketException(env, "Socket Closed");
                return -1;
            }
        }
    }

    nread = recv(fd, bufP, len, 0);
    if (nread > 0) {
        (*env)->SetByteArrayRegion(env, data, off, nread, (jbyte *)bufP);
    } else {
        if (nread < 0) {
            /*
             * Recv failed.
             */
            switch (WSAGetLastError()) {
                case WSAEINTR:
                    JNU_ThrowByName(env, JNU_JAVANETPKG "SocketException",
                        "socket closed");
                    break;

                case WSAECONNRESET:
                case WSAESHUTDOWN:
                    /*
                     * Connection has been reset - Windows sometimes reports
                     * the reset as a shutdown error.
                     */
                    JNU_ThrowByName(env, "sun/net/ConnectionResetException",
                        "");
                    break;

                case WSAETIMEDOUT :
                    JNU_ThrowByName(env, JNU_JAVANETPKG "SocketTimeoutException",
                                   "Read timed out");
                    break;

                default:
                    NET_ThrowCurrent(env, "recv failed");
            }
        }
    }
    if (bufP != BUF) {
        free(bufP);
    }
    return nread;
} 

And I found that on this line: "check if the socket has been closed while we were in timeout" - the method returns without releasing the possibly allocated bufP buffer. Well, I'm not good at C, but this seems to be a bug. And it's there in the latest jdk6 (31) as well, but it's fixed in OpenJdk7.
So I think this is the error that somehow causes an AV on win2003. Upgrading to jdk7 should help.

Testing a GWT/Mvp4g application in the JVM

I develop a client for a logistics system in GWT, using Mvp4g (currently GWT 2.3 with Mvp4g 1.3.1). The system has technically quite complex integration tests, where a J5EE based (Glassfish 2.1 + Seam 2.2) core application serves multiple WPF clients (with web services using WCF and Metro) and multiple GWT web clients. I use the same infrastructure to test the overall performance of the system. It might not have been the best decision, but I didn't want to use Selenium at that time, and HtmlUnit had (maybe still has) some issues with my application which I didn't want to sort out (although it should work), so I chose to instantiate my GWT application in the JVM, using mock views. The main application code is in the presenters anyway, so it should be easy to use from a JVM. Well, not that easy, but not a catastrophe.

Presenters and the EventBus

So, presenters should be instantiable without modification in the JVM. Any GWT UI related code should be in the views, that's not a big restriction. The EventBus itself is generated at compile time by Mvp4g, and I didn't want to use those generators, probably the generated code runs only in a browser, due to the use of Guice, but maybe I'm wrong :).

Anyway, the EventBus is quite simple to implement with reflection and dynamic proxies. What we have to do is basically:

• contain an instance of each presenter, and bind them to their views,
• maintain a list of presenters handling each event
• and delegate each event method invocation to those presenters, using the method name convetion.

So here is the code I use:

import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Vector;

import com.google.inject.Injector;
import com.mvp4g.client.annotation.Event;
import com.mvp4g.client.annotation.InitHistory;
import com.mvp4g.client.event.EventHandlerInterface;
import com.mvp4g.client.presenter.PresenterInterface;

public class EventBus<T extends com.mvp4g.client.event.EventBus> {
  
   Class<T> interfaceClass;
  
   T eventBus;
   @SuppressWarnings("unchecked")
   Map<Class<PresenterInterface>, PresenterInterface> presenters = new HashMap<Class<PresenterInterface>, PresenterInterface>();
   Map<String, EventDescriptor> events = new HashMap<String, EventDescriptor>();
   EventDescriptor initEvent;
  
   Injector injector;
  
   @SuppressWarnings("unchecked")
   public EventBus(Class<T> interfaceClass, Injector injector) {
       this.injector = injector;
       this.interfaceClass = interfaceClass;
       eventBus = (T) Proxy.newProxyInstance(EventBus.class.getClassLoader(),
               new Class [] { interfaceClass }, new EventBusInvocationHandler());
       for (Method method : interfaceClass.getMethods())
           if (method.isAnnotationPresent(Event.class)) {
               Event event = method.getAnnotation(Event.class);
               EventDescriptor eventDescriptor = new EventDescriptor();
               eventDescriptor.method = method;
               eventDescriptor.eventName = method.getName();
               eventDescriptor.targetMethodName = "on" + Character.toUpperCase(eventDescriptor.eventName.charAt(0)) + eventDescriptor.eventName.substring(1);
               for (Class<? extends EventHandlerInterface> cls : event.handlers())
                   eventDescriptor.handlers.add(getPresenter(cls));
               events.put(eventDescriptor.eventName, eventDescriptor);
               if (method.isAnnotationPresent(InitHistory.class))
                   initEvent = eventDescriptor;
           }
   }
  
   @SuppressWarnings("unchecked")
   public void bindView(Class<? extends PresenterInterface> presenterClass, Object view) {
       PresenterInterface presenter = getPresenter(presenterClass);
       presenter.setView(view);
       presenter.setEventBus(eventBus);
       presenter.bind();
   }
  
   public void init() {
       try {
           initEvent.method.invoke(eventBus);
       } catch (Exception e) {
           throw new RuntimeException(e);
       }
   }
  
   @SuppressWarnings("unchecked")
   public <T extends EventHandlerInterface> T getPresenter(Class<T> presenterClass) {
       T presenter = (T) presenters.get(presenterClass);
       if (presenter == null) {
           try {
               presenter = injector.getInstance(presenterClass);
               presenters.put((Class<PresenterInterface>) presenterClass, (PresenterInterface) presenter);
           } catch (Exception e) {
               throw new RuntimeException(e);
           }
       }
       return presenter;
   }
  
   public T getEventBus() {
       return eventBus;
   }
  
   class EventDescriptor {
       List<EventHandlerInterface> handlers = new Vector<EventHandlerInterface>();
       String eventName;
       String targetMethodName;
       Method method;
   }

   class EventBusInvocationHandler implements InvocationHandler {

       @Override
       public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
           if ("hashCode".equals(method.getName())) {
               return EventBus.this.hashCode();
           }
           EventDescriptor eventDescriptor = events.get(method.getName());
           for (EventHandlerInterface presenter : eventDescriptor.handlers) {
               Method presenterMethod = presenter.getClass().getMethod(eventDescriptor.targetMethodName, eventDescriptor.method.getParameterTypes());
               presenterMethod.invoke(presenter, args);
           }
           return null;
       }
   }
}

The Application class

In Mvp4g you define the entry point (or use the mvp4g built in entry point) to bootstrap the framework. In JVM, we create an Application class that does the initialization. While Mvp4g uses Gin, in the JVM we use Guice to do the injection stuff. There's nothing special to it, if you had a GinModule in Mvp4g, you can create a GuiceModule, and provide its injector to the EventBus above.

The Application contains and instantiates the GuiceModule, the EventBus and the mock views, and then calls EventBus#bindView method to give the views to the EventBus.

That's almost all, now you are able to instantiate your application, and play with it through the mock views. You might also directly call events from your test code.

GWT Service invocations

Well, if your application calls GWT services as well, you have to do some hacking about it. The GWT RPC implementation is not symmetric, which means that for example the Readers/Writers (Marshallers) are different on the client and server side. The stream written with a server side writer can only be read by a reader on the client side, not on the server side (classes: com.google.gwt.user.server.rpc.impl.(Client|Server)SerializaionStream(Writer|Reader) ).

Fortunately I wasn't the first to want to call a GWT RPC service from JVM, and there's a project called gwt-syncproxy. It can create sync and async proxies for you as well. I forked in my local workspace and added some functionality to support performance monitoring transparently (see below).

There was still a small issue. As I use this stuff from test code, I have to make sure that all async service invocations finish before I do my assertions in my tests. To achieve this, I extended gwt-syncproxy a little further, and added a little code that keeps track of invocations in each thread (and 'child'-threads), so that the test code can call a waitForInvocations() method before going on to the assertions.

I18n

When I added i18n to the application, my tests suddenly failed :) . It was because while in GWT, it creates and implementation of the message interface, in the JVM we have to replace it with something. Anyway, the localized strings are not so important during the tests, I don't call assertions textual content. However, there has to be an object with the interface. I love creating proxies, so here it is:

import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;

import com.google.gwt.i18n.client.LocalizableResource.Key;
import com.google.gwt.i18n.client.Messages.DefaultMessage;

/**
* A class to implement com.google.gwt.i18n.client.Messages derived interfaces when running in a JVM
*/
public class MessagesFactory {
  
   public static <T extends com.google.gwt.i18n.client.Messages> T createInstance(Class<T> cls) {
       return (T) Proxy.newProxyInstance(MessagesFactory.class.getClassLoader(), new Class [] { cls },
               new MessagesInvocationHandler(cls));
   }
  
   protected static class MessagesInvocationHandler<T extends com.google.gwt.i18n.client.Messages>
               implements InvocationHandler {
      
       protected Class<T> cls;
      
       public MessagesInvocationHandler(Class<T> cls) {
           this.cls = cls;
       }

       @Override
       public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
           if (method.isAnnotationPresent(DefaultMessage.class)) {
               return method.getAnnotation(DefaultMessage.class).value();
           }
           if (method.isAnnotationPresent(Key.class)) {
               return method.getAnnotation(Key.class).value();
           }
           return method.getName();
       }
   }
}

And in the Guice module, I have to bind it manually:

bind(MyMessages.class).toInstance(MessagesFactory.createInstance(MyMessages.class));

Performance logging

Once I got the taste of using and extending everything in strage ways, I also added performance monitoring on the client side for GWT RPC invocations.

To proxy requests on the client side, I found a solution by Nathan Williams. You can declare in your gwt.xml for which service interfaces you want to use the proxy and then use the bind method when creating the service to pass it an AsyncInvocationHandler that will be called before the actual invocation, and on success and failure. I also extended the gwt-syncproxy in my workspace to support these invocation handlers, and thus I can have my performance data from the integration and load tests too.

Seam, MDB, EJB and glassfish coming together

The situation seems to be quite complex but I think it can happen to anyone :) So I have a Glassfish 2.1 appserver with Seam 2.2GA. I use several Seam components in the web tier and they interact with stateless EJBs as well, because some functionality has to be accessible through a remote EJB interface.

Injecting EJBs to Seam components are quite simple, just use the @In annotation. Seam will notice that it's a session bean, and use the default jndi name to look it up, as configured in the components.xml:

    <core:init jndi-pattern="java:comp/env/YOUR-APP-NAME/#{ejbName}/local"/>

You can also inject Seam components to session beans, using the same @In annotation. Everything works fine, however, you have to define your local EJB references in the web.xml (on Glassfish), so that Seam can look them up from the web tier as well. Otherwise, local interfaces are not accessible from the web tier. The case is similar for Message Driven Beans (MDBs). Local interface references have to be declared.

If you just use @In annotations in your MDB (or any seam component invoked from the MDB), you'll get NameNotFoundExceptions for "java:comp/env/YOUR-APP-NAME/YOUR-EJB-NAME/local".

You have to declare your ejb reference in your MDB, and the easiest way to do it is to use @EJB annotations in your MDB class. This would be fine, but by default, the JNDI name will be "java:comp/env/your.message.driven.bean.full.ClassName/referenceVariableName", so you have to override it by using the @EJB(name="YOUR-APP-NAME/YOUR-EJB-NAME/local" annotation. By declaring this, Glassfish will make your local EJB interfaces accessible for the code running from the MDB.

Sun Glassfish and Oracle XE Distributed Transactions (XA)

So, we're using Glassfish v2.1.1, currently with Oracle 10g XE, running on a Java 6 runtime, using ojdbc14.jar. And we wanted to use distributed transactions :) We are also using JBoss Seam 2.2, but that's unrelevant, fortunately.

We configured the connection pool to use oracle.jdbc.xa.client.OracleXADataSource, and we disabled 'Return non-transactional connections' of course. When we tried to access the database from our Seam-connected web tier, the following exceptions came up:

[#|2010-10-01T12:09:12.383+0200|INFO|sun-appserver2.1|javax.enterprise.system.container.ejb|_ThreadID=22;_ThreadName=httpSSLWorkerThread-8091-1;|
javax.ejb.EJBException: nested exception is: javax.transaction.SystemException: org.omg.CORBA.INTERNAL: JTS5031: Exception [org.omg.CORBA.INTERNAL:   vmcid: 0x0  minor code: 0 completed: Maybe] on Resource [rollback] operation.  vmcid: 0x0  minor code: 0  completed: No
javax.transaction.SystemException: org.omg.CORBA.INTERNAL: JTS5031: Exception [org.omg.CORBA.INTERNAL:   vmcid: 0x0  minor code: 0 completed: Maybe] on Resource [rollback] operation.  vmcid: 0x0  minor code: 0  completed: No
 at com.sun.jts.jta.TransactionManagerImpl.rollback(TransactionManagerImpl.java:350)
 at com.sun.enterprise.distributedtx.J2EETransactionManagerImpl.rollback(J2EETransactionManagerImpl.java:1150)
 at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.rollback(J2EETransactionManagerOpt.java:433)
 at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:3801)
 at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:3619)
 at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1388)
 at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1325)
 at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:205)
 at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:127)

javax.ejb.EJBException: nested exception is: javax.transaction.SystemException: org.omg.CORBA.INTERNAL: JTS5031: Exception [org.omg.CORBA.INTERNAL:   vmcid: 0x0  minor code: 0 completed: Maybe] on Resource [rollback] operation.  vmcid: 0x0  minor code: 0  completed: No
 at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1395)
 at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1325)
 at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:205)
 at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:127)

[#|2010-10-01T14:20:10.992+0200|WARNING|sun-appserver2.1|javax.enterprise.system.core.transaction|_ThreadID=18;_ThreadName=httpSSLWorkerThread-8091-1;_RequestID=4131aa28-3401-4edd-bf90-54f605bcbb8e;|JTS5041: The resource manager is doing work outside a global transaction
oracle.jdbc.xa.OracleXAException
 at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1120)
 at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:249)
 at com.sun.gjc.spi.XAResourceImpl.start(XAResourceImpl.java:222)
 at com.sun.jts.jta.TransactionState.startAssociation(TransactionState.java:305)
 at com.sun.jts.jta.TransactionImpl.enlistResource(TransactionImpl.java:205)
 at com.sun.enterprise.distributedtx.J2EETransaction.enlistResource(J2EETransaction.java:607)
 at com.sun.enterprise.distributedtx.J2EETransactionManagerImpl.enlistResource(J2EETransactionManagerImpl.java:372)
 at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.enlistResource(J2EETransactionManagerOpt.java:144)
 at com.sun.enterprise.resource.SystemResourceManagerImpl.enlistResource(SystemResourceManagerImpl.java:98)
 at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:216)
 at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:337)
 at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
 at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
 at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
 at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108)
 at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:92)
 at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
 at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
 at org.hibernate.jdbc.AbstractBatcher.prepareSelectStatement(AbstractBatcher.java:145)
 at org.hibernate.id.SequenceGenerator.generate(SequenceGenerator.java:96)
 at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:122)
 at org.hibernate.ejb.event.EJB3PersistEventListener.saveWithGeneratedId(EJB3PersistEventListener.java:49)
 at org.hibernate.event.def.DefaultPersistEventListener.entityIsTransient(DefaultPersistEventListener.java:154)
 at org.hibernate.event.def.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:110)
 at org.hibernate.event.def.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:61)
 at org.hibernate.impl.SessionImpl.firePersist(SessionImpl.java:646)
 at org.hibernate.impl.SessionImpl.persist(SessionImpl.java:620)
 at org.hibernate.impl.SessionImpl.persist(SessionImpl.java:624)
 at org.hibernate.ejb.AbstractEntityManagerImpl.persist(AbstractEntityManagerImpl.java:220)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.seam.persistence.EntityManagerInvocationHandler.invoke(EntityManagerInvocationHandler.java:46)
 at $Proxy263.persist(Unknown Source)

When switching to ojdbc5.jar or ojdbc6.jar, a further detail came in the logs. The exception above doesn't explain the cause of the OracleXAException (in OracleXAResource.start). This seems to be fixed in later drivers, so in the logs we can see:

java.sql.SQLException: ORA-06550: line 1, column 13:
PLS-00201: identifier 'JAVA_XA.XA_START_NEW' must be declared
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored

Searching for this I got completely mislead. Forum entries say that there is no XA support in the XE version of 10g. Indeed, when I tried the whole stuff with Oracle 11g, it worked fine.

I wanted to try the stuff out myself to see how these XA handling looks like on the API level. I created a J2SE native app following the explanation and the code samples found here http://archive.devx.com/java/free/articles/dd_jta/jta-2.asp and here http://download.oracle.com/docs/cd/B14117_01/java.101/b10979/xadistra.htm. And they worked fine, on Oracle 10g XE, with ojdbc14.jar. So obviously, the 10g XE version does support XA.

But why is it then that under Glassfish, it tries to use some fancy JAVA_XA package? It's my habit to jump into the source code of anything I can find, and even read the Eclipse class view showing the JVM level code. So I found that from my J2SE stuff, a T4CXAConnection is returned from the OracleXADataSource, but somewhy under glassfish, it is OracleXAConnection. So why is this difference? Browsing the binary of the OracleXADataSource class I found two suspicious properties named useNativeXA and thinUseNativeXA. And indeed, under glassfish, they were both false, despite their default values of useNativeXA=false, thinUseNativeXA=true.

Searching for these properties I found some explanations here http://download-west.oracle.com/docs/cd/B19306_01/java.102/b14355/xadistra.htm#BGBBHCFC and here http://openesb-users.794670.n2.nabble.com/Definition-of-Oracle-XA-datasources-td4461045.html. So, the Glassfish admin console proposes a default value of the useNativeXA=false, and setting this property also disables thinUseNativeXA. So we either remove this property, or set it to true (the additional properties page of the connnection pool properties), and it'll work fine.

So, after all, I should just have read "Oracle® Database JDBC Developer's Guide and Reference" to be aware of this feature in the jdbc driver :) . Hope it'll help you guys.

Logically deleted entities in Hibernate Search

A recent project relies on Hibernate Search with a quite complex entity structure. We perform search on an entity that has several associated entities at multiple levels, and the search matches several text fields of the associated entities. Everything worked quite fine, although with 3.1.1.GA we had to workaround this bug: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-391 (already fixed in 3.1.2).

The problem came when we started logically deleting associated entities (using a "deleted" boolean field), because the search still matched for text values of the deleted entities, and there seemed to be no proper way to exlude these indexes (and I think there is really not, as the indexes in lucene are 'flat'). Fortunately the hibernate community helped me out: https://forum.hibernate.org/viewtopic.php?f=9&t=1003745

In my parent entities I filtered out these deleted entites from the collections, but this didn't make Hibernate Search drop the related indexes. The solution suggested in the forum was to actually remove the deleted entities from the owner collection, so actually breaking the relation between the entities. This might not be a solution in every case - you might need to keep the relation - but it helped me fortunately.