billroper: (Default)
Well, I added caching for another couple of classes today, which should reduce the memory footprint a bit further. Unfortunately, I am getting close to the point of diminishing returns.

The thing that's the real killer is a big doubly-linked list. Unfortunately, it's a big doubly-linked list that I really need.



Mar. 15th, 2017 10:58 pm
billroper: (Default)
Here I am, in the maze of twisty passages, all alike. Again.

When we open one of our models on the Java side, the memory usage in the JVM goes up by 200 MB. The heap dump says that the data associated with the model is only about nine MB.

Where the heck is the other 191 MB coming from? You'd think it would be easy to figure this out, but so far, not so much.



Mar. 14th, 2017 09:45 pm
billroper: (Default)
So we're studying heap dumps for our Java app to try to determine why our models are taking up as much space as they are. After staring down some of the models, I dropped in a bit of new code.

The model that I was working with -- one of the smaller ones, admittedly! -- shrank by almost 10% in its memory footprint. Larger models probably won't shrink quite as much, but it was a gratifying result.

Now to see if I can use this trick in some other places. :)
billroper: (Default)
Yesterday, I discovered that someone had (in error) created a new session object and left it to hang around forever.

Further investigation showed that had happened more than once, sometimes with big blocks of memory left attached to the session.

I have now exterminated those cases.

I also did a massive cut-and-paste operation to insert code that would throw an exception every time that an attempt to assert a transient lock failed. (Normally, it should wait a short period of time, the other lock should clear, and the operation should complete.) This was frustrating, but in the defense of the folks who wrote the code that wasn't throwing the exception, I hadn't added the quick-and-easy method of throwing the exception with all of the error information in it when they first started using the locks, so...

(Now, it's just call the constructor for the exception and pass in the lock object so that it can load up from the failure info. One line, nice and easy does it. This is because I am lazy and don't want to write this code more than once. :) )

All of this fell into the category of "more fun than human programmers should be allowed to have".


Dec. 15th, 2016 11:36 pm
billroper: (Default)
I am in a maze of twisty Java generics, all alike.
billroper: (Default)
So there's a class that I ported over from C++ to Java two years and a bit ago as part of this project. It works fine when you convert it from C++ and you can use it for a great many things successfully. However, when you try to edit it, the class falls to pieces.

I haven't really touched the class in any significant way in quite some time, so when I looked at it, I was pretty much horrified to find that I had managed to port it in such a way that there was the ported-from-C++ array of children for each member of the class and the Java array of children that was hiding in the new Java-side base class. Only one of these was populated and it was -- naturally -- the ported-from-C++ array, which meant that all manner of bad things are happening when I try to edit the data structures.

I am now in the process of removing the ported-from-C++ array (which I will only use during serialization because I have to be able to get the data back into the C++ side of the world) in favor of the Java-side array. This requires a lot of hand editing of code in this lovely 4300 line class.


Well, at least I've learned something in the past two years.
billroper: (Default)
My code appears to be working. The code that is downstream from it, maybe not. We're trying to get my code into the downstream environment to test it, but that is proving more difficult than it should.

billroper: (Default)
It took a while to get things straightened out, but I think I finally got my inner interface to successfully return a List of objects that implement the interface. Whee!
billroper: (Default)
I think this is clever. Your mileage may vary.

So in our Java Server implementation, when you authenticate with the Server, we create a Session object, put it into our Session Catalog, and pass back either a String that is the session ID or the actual Session object, depending on what environment you're operating in. The Session implements HttpSessionBindingListener so that it will get a callback when the bound Session is being destroyed. When we get the callback, we take the Session out of the Session Catalog, which should, in theory at least, be the last strong reference to the Session object.

We may have opened a number of objects of different types on behalf of the Session. Each type of object has its own Catalog, which -- in this case -- is stored in a static WeakHashMap where the Session is the key and the Catalog of objects opened on behalf of that Session is the stored value. So when the Session is headed off to the Garbage Collector, we'll also garbage collect the Catalog of objects that was associated with it in the WeakHashMap.

The advantage of this approach is that when I add a new type of object to the system that I want to open on behalf of the Session, I don't have to change the Session object. I just have to create a static WeakHashMap that holds all of the Catalogs for the new object type. This also puts all of the Catalog information in the place where I'd like it to be, given the general architecture of the Java Server. In this case, the Catalog classes are all inner classes inside of the object class that they hold.

As I said, I think it's clever... :)
billroper: (Default)
It was one of those days, at least in some respects.

I had fixed a number of bugs and ported a bunch of code fixes from C++ to Java. But there was an intractable bug that turned out to be due to corrupted data in one of my classes. In fact, what I was seeing was that the child class was pointing back to the wrong parent as its owner.

So it seemed that something had to be wrong with either the copy constructor or the clone() method in my Java code. And I stared at them. And stared at them.

And about a half-hour after I'd started staring, I suddenly realized that I was setting the parent of the children incorrectly when I was cloning the parent class. Because you don't set the parent to this, you set the parent to the newly created parent.

I could easily have stared at that for a lot longer before figuring out that I had typed something witless.


On the other hand, the bug appears to have gone away now.
billroper: (Default)
In one of those things that surprised me, it appears that there's no one in our group who really understands how to configure a WebLogic WorkManager -- either the one that is integrated into JDeveloper or the standalone version. The problem is that the default WorkManager will run wild if we hand it a large consolidation to run and will gleefully consume all of the available resources on the machine, so I really need to put some constraints on the process. And while I could try to put those in front of the WorkManager calls, the right place to put them is into the running WorkManager.

If only I knew how to do this. And I applied an awful lot of Google-fu to the process, but -- although there are many places that reference the constraints and sort of talk about how they work -- there isn't really a good set of examples for how to implement them.

Eventually though, I found enough breadcrumbs that I was able to figure out how to get this working in JDev. And once I figured that out, I was able to translate things out into the standalone WorkManager.

I'm sure there are a number of improvements that we can make to my configuration.

But this will do for today.
billroper: (Default)
For testing, I've written a Java implementation of MFC serialization. We won't be shipping it, but it allows us to test things for now.

The problem is that Java and C++/MFC have very different ideas about how to handle various data items. For example, Java would like IDs to be immutable and shared among different objects that reference the same object by that ID. C++ doesn't want to share those IDs, because reference counting, well, not so much. So if you edit something in Java, the course of least resistance is to share the IDs, but then you need to make them all unique when you serialize them so that they work the way that C++ expects.

This turns out to be more difficult than you might think.

billroper: (Default)
It's being like that today. I had intended, at some point, to rewrite the entity data handling on the new Server architecture. Sadly, the current entity data handling is causing our consolidation to run out of memory, so I really need to rewrite it now.

The good news is that -- after a brief false start -- I have figured out how to structure things so that they work in a fashion that's similar to everything else on the Server that requires locking. The locking itself still needs some work for a clustered environment, but that's something that can be replaced in one swell foop once we get everything else straightened out. Since we're not yet running clustered, the current locking will do for the moment.

The amusing thing is that I have finally found a place where using a WeakHashMap is the right answer. :)

I'll still be cleaning this up tomorrow morning, but there is a non-train light at the end of the tunnel.
billroper: (Default)
Yesterday's intractable problem at work certainly appears to be a Java bug.

Take, for example, the case where you have wrapped an InputStream inside an InflaterInputStream -- in this case, an InputStream that is actually an OracleBlobInputStream, although it fails equally well should you be connecting to SQL Server. This should, in theory, allow you to read a compressed BLOB out of a database table. And as long as you are calling the read() method with no arguments and reading the stream one byte at a time, it does.

Reading the stream one byte at a time is not the most efficient way of handling it though, especially when you know that the next thing that is on the stream is a byte array of a particular length. In that case, you would like to use the read( b, off, len ) method on the stream so that you can read multiple bytes from the stream in one operation. In fact, you should be able to set the offset to 0 and the length to the length of the array and you can do this in one beautiful call.

Most of the time.

There's something that's busted inside the native code implementation of Inflater.inflateBytes so that it can't manage to make this work all of the time. I strongly suspect that it hits a boundary between blocks that the Deflater wrote out and just stops in mid-read so that a call that should return, say, 53 bytes of data only returns 10 bytes. The rest of the data is there, but your read( b, off, len ) method simply isn't going to let you have it.

I apparently ran into this problem in June of 2014 and coded around it at the time by reading the byte array one byte at a time. But then I forgot about it and wrote some more code that gleefully tries to read the byte array in one operation.

And it works.

Most of the time.

Sadly, most of the time is not nearly good enough.

*mutter* *mutter*
billroper: (Default)
Vacation being over now, it was time to get back to work.

Today, I fixed a flawed Comparator that I'd copied from the C++ code and not modified quite correctly. And I refactored some other code that was failing so that it behaves in a more Java-like fashion and no longer fails in the way that it was failing before.

Tomorrow, who knows?
billroper: (Default)
So I'm working on setting up an API for some of the user-interface guys to use to access our data on the server. This frequently involves them sending in an assortment of Strings that uniquely identify some object so that we'll open it and cache it for them to access repeatedly. And when they want to access it again, they'll send me that sequence of Strings again and I'll find what they're looking for.

It strikes me that life would be much more efficient if I gave them a unique handle to the object that they've opened so that they can reference it by the single String that would contain the handle. I've sent them an e-mail proposing this.

We'll see what they say in the morning.
billroper: (Default)
Having been tasked to generate some JSON in Java, I went out looking for tools, because I hate dealing with the trivia of punctuation. :)

I found the JsonWriter which generated correctly formed JSON.

Slowly. Very slowly. As in, it took about seven seconds to format 18,000 cells of data based on my rough count as it ran in the JDev debugger.

Not surprisingly, folks were not exactly thrilled with that report.

So I did some more research today. It turns out that JsonWriter constructs a JSON structure in memory, much like DOM does for XML. If you've got an itty-bitty structure, this is just fine. If you have large structures, not so much.

Happily, there is a streaming JSON API via the JsonGenerator interface. It plunked out the same 750K of data in less than a second. The limiting factor no longer appears to be the speed with which I can generate the JSON, but rather the speed with which I can retrieve the data and convert it from numeric to string format.

I like streaming APIs...
billroper: (Default)
So one of the other programmers in the group thought he'd found a bug in some of my code. I asked if he could set up a screen share so that I could watch him step through the failing code. And after a little bit of research, I recoiled in horror.

In some code that I had written fairly early in my Java development, I had pretty grotesquely mishandled the immutable DateTime class from the Joda Time package. I'd called a method on a DateTime that would return a new DateTime, but I hadn't assigned it to anything, with the net result that it pretty much did nothing. Since I wanted it to do something, that was a bad thing. Apparently, I'd assumed I was modifying the immutable class.

Not so much.

So I fixed that and then went on a snipe hunt through the rest of the code to find the places where I'd used the library badly. There were only a couple of actual bugs, but there were several places that I could -- and now have! -- improved things substantially.

I guess the good news is that I'm learning. :)
billroper: (Default)
I was writing a new interface layer in Java 7 today where I'm being handed a bunch of key/value pairs to use to look up and return various items of data.

So I wrote an immutable class with a constructor that processes the key/value pairs and loads up the keys into the correct locations in the object.

But I need to sort the queries so that all of the Key A queries are grouped together and all of the Key B queries are grouped together within the Key A queries. So I wrote a compareTo method for the immutable class that works through all of the data elements in the class, starting with Key A, then Key B, working down through Key N. Then I tossed the keys into a TreeSet, which happily sorted them so that when I walk through the elements, all of the matching values for Key A come first, all of the matching entries for Key B within Key A come next, and so on.

Then I created a class that would load an instance of the big in-memory dataset that Key A refers to. And I wrote a moderately long switch/case statement for each of the different kinds of data that I might return from the big dataset based on the original class that loaded up all of the key/value pairs.

I think this might work.

Of course, I still have to test it... :)
billroper: (Default)
Wonderful. I can't check his test case in my code, because when I try to save the file with my changes in it, the serialization code that he wrote for some other stream in the document writes stuff that can't be read by the old code which is the only place that I can reliably look at the output data.

Meanwhile, his test case continues to fail after all of my changes.



billroper: (Default)

July 2017

2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jul. 24th, 2017 12:49 pm
Powered by Dreamwidth Studios