Exception handlers often broken in converted Android code

input:

java -jar procyon-decompiler-0.5.15.jar f.class > 123

output:

    // java.lang.IllegalStateException: Expression is linked from several locations: Label_0243:
        //     at com.strobel.decompiler.ast.Error.expressionLinkedFromMultipleLocations(Error.java:27)
        //     at com.strobel.decompiler.ast.AstOptimizer.mergeDisparateObjectInitializations(AstOptimizer.java:2386)
        //     at com.strobel.decompiler.ast.AstOptimizer.optimize(AstOptimizer.java:214)
        //     at com.strobel.decompiler.ast.AstOptimizer.optimize(AstOptimizer.java:42)
        //     at com.strobel.decompiler.languages.java.ast.AstMethodBodyBuilder.createMethodBody(AstMethodBodyBuilder.java:214)
        //     at com.strobel.decompiler.languages.java.ast.AstMethodBodyBuilder.createMethodBody(AstMethodBodyBuilder.java:100)
        //     at com.strobel.decompiler.languages.java.ast.AstBuilder.createMethodBody(AstBuilder.java:668)
        //     at com.strobel.decompiler.languages.java.ast.AstBuilder.createMethod(AstBuilder.java:567)
        //     at com.strobel.decompiler.languages.java.ast.AstBuilder.addTypeMembers(AstBuilder.java:493)
        //     at com.strobel.decompiler.languages.java.ast.AstBuilder.createTypeCore(AstBuilder.java:460)
        //     at com.strobel.decompiler.languages.java.ast.AstBuilder.createTypeNoCache(AstBuilder.java:140)
        //     at com.strobel.decompiler.languages.java.ast.AstBuilder.createType(AstBuilder.java:129)
        //     at com.strobel.decompiler.languages.java.ast.AstBuilder.addType(AstBuilder.java:105)
        //     at com.strobel.decompiler.languages.java.JavaLanguage.decompileType(JavaLanguage.java:55)
        //     at com.strobel.decompiler.DecompilerDriver.decompileType(DecompilerDriver.java:270)
        //     at com.strobel.decompiler.DecompilerDriver.main(DecompilerDriver.java:114)
        //  

I don't suppose you would happen to have the original, unobfuscated class, would you? Or the original source? :)

I get this a lot too but only for dex2jar produced code. Just to check though I used another tool, Dare which again converts a dex into class files. Exactly the same result from procyon, Expression is linked from several locations so the code seems to be valid jvm code as far as I can tell. Would be nice to have this work, really nice but other Java decompilers are failing at this, including cfr.

yeah, Marius, you got it right, the java classes were produced by the d2j but, isn't it a beauty of the reality to break the rules? ie to do things that others takes as rule "just not possible because not possible?" :)

Oh it is, it is. De-compiling closed source Android apps to see how they work is so much fun. I played around with whatsapp quite a bit so I can make an open source version.

Maybe Mike can help us out here, I've tried looking into the code but it's not so easy to determine what's wrong.

my words was not about 3rd party closed app, but to the fact that other decompilers can't do it - that is the aim, the app was taken as a random one.......

Oh I see now. I'm interested in some particular ones but that's for Mike to decide whether he wants to beat his co-worker that's developing cfr :)

Looks like dex2jar produces some really nasty exception handler tables. This definitely isn't going to be an easy fix.

Mike I've attached the class file produced by dex2jar and dare for a case where procyon fails in a similar way. Maybe the one produced by Dare is better.

Couldn't attach here for some reason so: http://www62.zippyshare.com/v/1976085/file.html http://www62.zippyshare.com/v/27137416/file.html

Also here's a idea which won't fix the issue entirely but it would be good enough for me. If a class method can't be decoded and it has some try/catch blocks go back and decode it but ignore all the try/catch and finally.

If this is not too hard to do it's good enough for me.

I can then add the try/catch code myself if needed but I'd much prefer this to no method decoding at all.

@mike Probably a good idea is to report to dex2jar team your concerns regarding exc handlers?

Alex if dare produces similar results ( and it seems it does ) I'm not sure what dex2jar can do about it. Dex2jar is also not a team, just like Mike it's more of a one man show :). Mike when you have the time could you look if the code produced by Dare is in any way better. Dare even has a flag that's supposed to make it easier for decompilers and I've used it. From what I could tell the dare code does look a bit better.

More in Issue #120.

Issue #120 was marked as a duplicate of this issue.

Marius, may I suggest using Krakatau instead? It's designed handle stuff like this just fine.

Uncaffeinated I did try Krakatau but didn't have much luck with it either. I find it harder to use also. Took me a while to figure out it wants all the classes referenced by the ones to decompile so I had to add the android.jar to the path. Also wasn't clear what the path separator was so as to add multiple jars.

This seems like a big difference, Mike can procyon use extra objects in the class path to provide better results ? Like in this case would having the android framework jar help since a lot of classes import classes from there ?

Anyway the more the better. Often one compiler fails and another succeeds and I'm sure Mike will get around to fixing this, I only wish commercial software would have bug fixing like procyon does.

Right now though with the dex2jar generated jar I'm afraid Krakatau behaves even worse than procyon, it just hangs. I'll report the issue.

If you found a bug or have suggestions for where the documentation should be improved, feel free to report it over at https://github.com/Storyyeller/Krakatau. It definitely shouldn't be just hanging.

OK so after testing Krakatau it seems dare actually produces better code from an Android dex or so it seems. Below are 2 jars from Dare and dex2jar made from the same APK:

http://www25.zippyshare.com/v/73555638/file.html

http://www25.zippyshare.com/v/34367401/file.html

These work with procyon but some methods give the errors above ( the duplicate label and the index problem ) . Duplicate label is also a problem for other decompilers but the index one is not.

This seems like a big difference, Mike can procyon use extra objects in the class path to provide better results ? Like in this case would having the android framework jar help since a lot of classes import classes from there?

Procyon will do its best to try to resolve dependencies with whatever information it has available. This includes probing the system and bootstrap classpaths (including your CLASSPATH environment variable). It also maintains a memory of where it has found classes before, and may probe the file system according to package structure. For instance, if you were to run procyon ./C.class, and C.class is the type a.b.C, Procyon will remember that directory as a known location for the a.b package. Then, if it needs to find the class a.X, it'll try searching one level up in the file system.

Procyon will always attempt to locate referenced classes as they are needed, but it should be able to produce reasonable output even if it fails to load every other class. In theory, the only classes it must be able to resolve are:

1. The class you are trying to decompile;
2. The class java.lang.Object.

You will get the best results when all referenced classes can be resolved. You may see artifacts due to failed class resolution, most notably:

1. There may be redundant casts at method call sites if method overload resolution cannot be performed (e.g., because the target class could not be loaded).
2. Missing inner classes (especially anonymous inner classes) will result in some artifacts being left behind.
3. Variable types may not be as specific as they could be. If Procyon has to figure out a common supertype for two classes, but it can't resolve all the base classes and interfaces, it may end up picking java.lang.Object.

But in the case of the errors here, they all seem to be related to the exception handler structure. The DEX to JAR converters seem to produce a lot of overlapping handlers, which I don't handle very well at the moment.

I have tried both d2j and dare to reproduce the class files and I also get this error. perhaps I could attach my sample class file produced by dare and you could check if it does the same thing as d2j does.

dare - https://dl.dropboxusercontent.com/u/16422464/classes/dare/DatabaseHelper.class

d2j - https://dl.dropboxusercontent.com/u/16422464/classes/d2j/DatabaseHelper.class

Here is some of my research TryCatchTest.zip

I took TryCatchTest.java from Krakatau and made the following:

1. Unobfuscated: javac->dx->dex2jar->procyon
2. Obfuscated: javac->zkm->dx->dex2jar->procyon

Both provides huge exceptional tables. Even without obfuscation Procyon produces damaged sources (while fernflower and jd-gui does not at all)

hi guys, I am the author of dex2jar, here is why, the exception table is huge.

the following code is valid in dex, from issue 109 of dex2jar https://code.google.com/p/dex2jar/issues/detail?id=109

L0:
 //...
L1:
//...
L4:
//...
return a2;
L2:
// ...
L5:
// ...
L6:
return a2;

L0 ~ L1 > L2
L4 ~ L5 > L6

because dalvik has different control flow to jvm, only a throwable op goes to the exception handler. but in jvm, any op in the exception table goes to the exception handler.

so, in jvm, a path (L0,L2,L6,return a2) is created, and cause a validationException mention that a2 is not initialized.

to fix this, i cut the exception table into pieces. Its a simple but stupid change.

any suggestion?

Interesting. Krakatau effectively does the same thing internally in the hopes of simplifying the code. But if the input can't easily be expressed with Java control flow, then the output will be really messy no matter what you do.

Hi pxb1988. First of all thank you for dex2jar, together with Procyon or Krakatau it proved very useful in understanding how some Android apps work. I'm not the most qualified to comment here due to my limited understanding of this but what I did notice is that Dare seems to take a slightly different approach to it. Not sure if it's better, if it's correct but certainly seems different. There are some attached files from both Dare and D2J.

Hi Mike

seen this issue on the latest .25 build:

    //    Exceptions:
    //  Try           Handler
    //  Start  End    Start  End    Type                 
    //  -----  -----  -----  -----  ---------------------
    //  91     117    128    721    Ljava/lang/Exception;
    //  91     117    318    328    Any
    //  117    125    128    721    Ljava/lang/Exception;
    //  117    125    318    328    Any
    //  130    189    318    328    Any
    //  195    200    701    712    Ljava/lang/Exception;
    //  195    200    318    328    Any
    //  223    249    128    721    Ljava/lang/Exception;
    //  223    249    318    328    Any
    //  249    275    128    721    Ljava/lang/Exception;
    //  249    275    318    328    Any
    //  275    307    128    721    Ljava/lang/Exception;
    //  275    307    318    328    Any
    //  307    315    128    721    Ljava/lang/Exception;
    //  307    315    318    328    Any
    //  328    355    128    721    Ljava/lang/Exception;
    //  328    355    318    328    Any
    //  363    397    128    721    Ljava/lang/Exception;
    //  363    397    318    328    Any
    //  400    427    128    721    Ljava/lang/Exception;
    //  400    427    318    328    Any
    //  435    451    128    721    Ljava/lang/Exception;
    //  435    451    318    328    Any
    //  451    478    128    721    Ljava/lang/Exception;
    //  451    478    318    328    Any
    //  478    574    128    721    Ljava/lang/Exception;
    //  478    574    318    328    Any
    //  574    582    128    721    Ljava/lang/Exception;
    //  574    582    318    328    Any
    //  585    598    128    721    Ljava/lang/Exception;
    //  585    598    318    328    Any
    //  601    628    128    721    Ljava/lang/Exception;
    //  601    628    318    328    Any
    //  636    643    128    721    Ljava/lang/Exception;
    //  636    643    318    328    Any
    //  646    674    128    721    Ljava/lang/Exception;
    //  646    674    318    328    Any
    //  677    683    128    721    Ljava/lang/Exception;
    //  677    683    318    328    Any
    //  683    693    128    721    Ljava/lang/Exception;
    //  683    693    318    328    Any
    //  703    709    318    328    Any
    //  712    718    318    328    Any
    // 
    // The error that occurred was:
    // 
    // java.lang.IllegalStateException: Expression is linked from several locations: Label_0200:
    //     at com.strobel.decompiler.ast.Error.expressionLinkedFromMultipleLocations(Error.java:27)
    //     at com.strobel.decompiler.ast.AstOptimizer.mergeDisparateObjectInitializations(AstOptimizer.java:2592)
    //     at com.strobel.decompiler.ast.AstOptimizer.optimize(AstOptimizer.java:235)
    //     at com.strobel.decompiler.ast.AstOptimizer.optimize(AstOptimizer.java:42)
    //     at com.strobel.decompiler.languages.java.ast.AstMethodBodyBuilder.createMethodBody(AstMethodBodyBuilder.java:214)
    //     at com.strobel.decompiler.languages.java.ast.AstMethodBodyBuilder.createMethodBody(AstMethodBodyBuilder.java:99)
    //     at com.strobel.decompiler.languages.java.ast.AstBuilder.createMethodBody(AstBuilder.java:756)
    //     at com.strobel.decompiler.languages.java.ast.AstBuilder.createMethod(AstBuilder.java:654)
    //     at com.strobel.decompiler.languages.java.ast.AstBuilder.addTypeMembers(AstBuilder.java:531)
    //     at com.strobel.decompiler.languages.java.ast.AstBuilder.createTypeCore(AstBuilder.java:498)
    //     at com.strobel.decompiler.languages.java.ast.AstBuilder.createTypeNoCache(AstBuilder.java:140)
    //     at com.strobel.decompiler.languages.java.ast.AstBuilder.createType(AstBuilder.java:129)
    //     at com.strobel.decompiler.languages.java.ast.AstBuilder.addType(AstBuilder.java:104)
    //     at com.strobel.decompiler.languages.java.JavaLanguage.decompileType(JavaLanguage.java:59)
    //     at com.strobel.decompiler.DecompilerDriver.decompileType(DecompilerDriver.java:301)
    //     at com.strobel.decompiler.DecompilerDriver.main(DecompilerDriver.java:127)
    // 
    throw new IllegalStateException("An error occurred while decompiling this method.");
}

I guess its same but in a case you'd like to take a look I could PM you the class file Alex

I ran into this problem with Android classes as well. I think I have traced it to ensureDesiredProtectedRanges (called by pruneExceptionHandlers() in compilertools.com.strobel.decompiler.ast.AstBuilder)

(The code below has some of my comments and debug statements in it. Note that I commented out the 'handlers.set()' call to test the 'fix'.) This is complicated by the fact that I don't really understand that 'ensureDesiredProtectedRanges' is supposed to do. The effect is that ensureDesiredProtectedRanges is messing up the length of the try block by extending it to the instruction before the catch block.

Commenting out the handlers.set() fixes the 'Expression is linked from several locations' problem for me (not the IndexOutOfRange exception version of the problem).

private void ensureDesiredProtectedRanges() { final List<ExceptionHandler> handlers = _exceptionHandlers;

    for (int i = 0; i < handlers.size(); i++) {
        final ExceptionHandler handler = handlers.get(i);
        final InstructionBlock tryBlock = handler.getTryBlock();
        final List<ExceptionHandler> siblings = findHandlers(tryBlock, handlers);  // List of handlers with same tryBlock
        final ExceptionHandler firstSibling = first(siblings);
        final InstructionBlock firstHandler = firstSibling.getHandlerBlock();
        final Instruction desiredEndTry = firstHandler.getFirstInstruction().getPrevious();

        for (int j = 0; j < siblings.size(); j++) {
            ExceptionHandler sibling = siblings.get(j);

            if (handler.getTryBlock().getLastInstruction() != desiredEndTry) {
                // Last instruction of exHndlr try block is not the firstInstruction of the firstHandler
                final int index = handlers.indexOf(sibling);
                System.out.println("  eDPR "+handler+" tryEnd != "+desiredEndTry //RAF
                  +"\n    "+j+"/"+handlers.size()+" hndlrs, "+index+"/"+siblings.size());
                org.ribo.jDecomp.rLog.append("eDPR "+this._context.getCurrentMethod().getName()
                  +" "+_context.getCurrentType().getName()+" "+sibling+"\n");

                if (sibling.isCatch()) {
                    // This is a try/catch (not try/finally)  //RAF
                    //handlers.set(  
                    //    index,
                    //    ExceptionHandler.createCatch(
                    //        new InstructionBlock(
                    //            tryBlock.getFirstInstruction(),
                    //            desiredEndTry
                    //        ),
                    //        sibling.getHandlerBlock(),
                    //        sibling.getCatchType()
                    //    )
                    //);
                }
                else {  
    ...

What fix to the list of exceptions is this method supposed to do? Is it supposed to merge two handlers or adjust the range of the try block? In any event I'm suspicious of the:

desiredEndTry = firstHandler.getFirstInstruction().getPrevious();

statement. desiredEndTry is an instruction preceding the catch handler code block -- changing the end of the try code block to be something before the catch block code. In the Android code I'm decompiling, the catch block doesn't directly follow the try block code. This the try block ranges of code of two different try blocks to overlap.

So, what is ensureDesiredProtectedRanges intended to fix in the list of exception handlers? -- or what assumptions is it making about the relationships between trys and catches?

I've instrumented the decompiler code so I can see what is going on, so I can test things out.

I think the best answer here is to just use an android decompiler instead of dex2jar + a java decompiler. Converting to a jar first will always be a lossy process.

I'm not so sure using dex2jar or similar + java decompiler can't have perfect or almost perfect results. Of course the Java decompiler probably has to do some tricks it wouldn't normally do but it should work.

The problem with using an android to java decompiler is that I don't know of one that's free or at least reasonably priced. JEB costs too much money for me. I'd like it open source or maybe I'd pay like $100 it were really, really good.

I haven't really investigated Android compilers since I was never interested in Android, but if worst comes to worst, you could always just modify Krakatau to read dex files. Sadly, I'm probably the best person to do that but I don't have the time or interest any more.