I haven't posted for a long time but I've got a good excuse. Really I do. I haven't posted for four months, but I've only been idle from WeldAE for only three months. I worked on it pretty heavily until about June 26th. After that the amount of responsibilities I had really wouldn't allow me any free time for side work. From June 26th until today I've:

Searched for and purchase a big SUV out of state which required me to fly there, pick it up and drive back
Ripped up all the carpet in my house so new carpet could be laid. This shut down my network for two weeks alone and truth be told, I still haven't really recovered yet.
Finished a 300 square foot bonus room as my new office. This room is all 36 degree angles and was a joy to insulate, sheet rock and paint.
Turn my old office into a guest room
Paint and furnish a new nursery in what was my wife's office
Spend a week in the hospital when our twin girls arrived
Wake up every two hours to help feed the twins

In the last few days, things have settled down a good bit and I'm back to a point where I have a little bit of free time. So what's been going on with WeldAE? The reason I didn't post more in June was that I had branched the server side to experiment with a big change I've been wanting to make for a long time. To be honest, I can't even remember what caused the branch but I do remember I couldn't take the code much further without some sort of big change on the server side. I decided to go whole hog and I'm glad to say as of a few minutes ago it was a complete success. I've still got to convert all my forms over to the new system and I've got to test and patch the code pretty heavily for PHP4, but I'm 100% this branch will go forward as the new development head and I won't revert back to the old system.

So what did I change? A lot when looking at the server code. When looking at the code written for WeldAE, the change is pretty small. Not to say it isn't a big deal since it will be much nicer to write code for WeldAE with the new changes. So here is a before and after example code that runs when a button is clicked and we want to change the width of the form:

Before

      button_onclick(){
         $this->form->set('width', 200);
      }

After

      button_onclick(){
         $this->set('width', 200);
      }

Before, $this referred to the script object. This object was an extension of a base script class and was loaded dynamically before each event was run. Since the class was being loaded at runtime it was easy to make sure the correct script class was loaded before any code could be run. Now the script runs as a class that extends the base form class. Since this object is being directly serialized in the session, I have to know all the form classes I need to include in my script before I can un-serialize my session object. Currently I'm using the __autoload PHP5 magic function to load the classes as the session is un-serialized. This is one of the bits of code I've got to change to make the new system work with PHP4.

I've always wanted WeldAE to work like this but there is some pretty crazy dynamic class stuff happening behind the scenes to make all this happen. I avoided it all this time because I didn't think I could make it happen without making the system much slower and more complex. It took about 40 hours of work, but I think I've made the system much more simple while at the same time not slowing it down any. This also underscores why there isn't more documentation for WeldAE yet. I keep changing the fundamental structure to much to make it worthwhile. That said, I don't see any major changes left. Going back to where I was in June, I think WeldAE needs lots of polish to get it ready for an Alpha release.

Just a quick update. I hope to add more later when I have more time.

I've really fixed a lot of small polish items with Weld, WeldAE and the Form Designer application. Some examples are correctly hinting forms so only the main form is displayed in the task manager. Along the same line, the parent form is now defaulted to the main form so setting a new form to "center on parent" at the very least will center on the main form if the form is launched from a script and no parent can be determined. One non-polish item I've worked on is the "Save As" dialog layout for the Form Designer. Check out the screen shot. Doesn't look like much, but there is a lot of thought for how the entire development system will eventually work behind the fields on that screen.

I started a new job last week and I don't know how it will affect the pace of WeldAE development. The initial effect was good and I got a lot done in the week prior to starting the new position. I knew that depending on the demands of the new job I might not have time to code for several weeks and I wanted to get as much done before I started. I didn't get much done during my first week, but I don't think that had anything to do with the new job and I think after I settle in I'll actually have more time to devote to WeldAE. My previous job was as a consultant and that required I travel a good bit. The travel was balanced by the fact that I could work out of a home office. My new job requires no travel but I work in an office which takes a surprising amount of time out of your day. I think the change will be a net positive for development.

So what did I get done since my last post? The server side PHP code has received a lot of attention. I've cleaned up all the remaining places I was calling the database directly and now everything goes through the WeldAE database object. I cleaned up the separation of Application, Form, Control and Event properties. Each of these objects now has three property types.

Object properties are directly associated with the object and are not part of the object's synced property array. These properties are not needed client side and don't sync. Even though developers can create custom object properties, they are mainly used by WeldAE and used to store meta data used to tie the object back to the database. Some property examples are createdate, modifydate, createuser, formid.
Normal properties are stored in an object's property array and are synced back and forth between the server and client. These properties hold all the important information about the object and the developer will uses these properties to set the width, height, color, etc. of an object.
Temporary properties are used to hold custom properties for temporary use and sync down from the client but not back up. They are stored in the property array of the object and are just like normal properties except they begin with an "_". The one way sync nature keeps the amount of data going back and forth between the client and server down while allowing meta data to be easily attached to an object.

So why have three property types? Well, the Object and Normal properties are inherently needed for the system to work. I could probably make all the Object properties Normal properties, but there would be some serious namespace problems and a massive amount of useless data would go back and forth. If anything, Temporary properties are the easiest to do without which is supported by the fact that I just added them to Weld last week. The need arose from the Form Designer application's need to save forms back to the database. I have tons of meta data I need to store about each object just to make the application able to design forms and I had to have a system for identifying what was a "property" and what was just meta data. I was just prefixing all my meta properties with "designer_". However, this seemed messy and I noticed that it was resulting in a lot of sync traffic that wasn't needed by the client. I couldn't just use an Object property because I needed the client to set the property directly without a round trip call to the server. Temporary properties solved all my problems in a clean way that was easy to implement.

The title isn't exactly correct, but speed measurements never are. I noticed that when clicking on controls in the form designer the entire screen would flash. I'd noticed this before but never investigated what might be causing it. I quickly discovered that this was caused by the server echoing client property settings back to the client when an event is run. I knew this was happening but I didn't realize how much overhead it was causing. I hadn't implemented the ability to set a property as "non-dirty" on the server side. I did this from the beginning on the client because it won't run at all without it. However, on the server side having all property sets() be dirty just resulted in bloated XML. I wasn't that worried about it for now but what I didn't think about was that this was causing the client to re-calculate a lot of properties to make sure they didn't need updating. Width and Height properties are especially bad because it was causing my flicker problem. Here is what was happening:

Client gets the form from the server. Most control widths are relative to the width of the screen so the server doesn't know the explicit value
The client calculates the specific widths of all the controls and marks these properties as dirty so the next trip to the server they will be sent
The user does something to cause a server side event to fire and all dirty properties are sent back to the server
The server processes all the properties as dirty since I was lazy and hadn't coded it to not mark properties coming from the client as dirty
The server sends all the dirty properties back to the client. This is all the properties the client sent plus any that changed.
The client processes all the properties and has to calculate all the relative properties to make sure they didn't change.
Because the server is moving other stuff around, the system thinks the values don't match and it moves everything around a bit then moves it back causing the flicker

Once I saw how bad the issue was I spent 30 minutes fixing it. The return XML size is about a third what it was before. Add to that the fact the client doesn't have to process all those complex events and the speed of Weld when using the Form Designer application is about twice as fast. Of course, the Form Designer is a very complex Weld application so I don't expect a normal application would have seen as much improvement, probably more like 10%.

This issue has also underscored how the application itself can cause a lot of XML spam. When a user clicks on a control I'm looking that control type up in the database and building the left side property panel to include all properties for that control type. To do this I'm just looping through all the property controls and hiding them. I then loop through all the properties for the control type clicked on and setting those property controls to visible. This results in all property controls having at least one property sent back to the client even if the property is already correct on the client. I could change the way my code works so I'm not double setting properties and potentially causing this problem, but I'm toying with the idea of keeping a shadow copy of the client property state and then using that to compare with the property state of the server to send back to the client.

It sounds very memory wasteful but it would also solve some API issues I've had with PHP4. In PHP5 I can setup setters and getters to track which properties have been changed that isn't possible with PHP4. To get around this I require all properties to be set using:

    $control->set("width", 200);

I'd rather it work more like:

    $control->width = 200;

I can do the later with PHP5 but I want to maintain a single API style. I need to find out how large my objects are in PHP before I can really even make a guess. The serialized session object is running ~200K for the Form Designer application. I don't know if serializing the data expands or shrinks the data size compared to the size as an object in PHP.

Control selection is now bug free and feature complete in the Form Designer application. This may sound simple, but selection is a complex problem given that state has to be saved per form and that each form can have multiple controls selected. Controls can selected by clicking on them and also by using the drop down control on the property inspector. If is currently down while a control is selected, that control is added to the selection list and if not, all the controls in the selection list are removed and only the new control is added. The last selected control is saved per form so that when the user switches between forms, the last selected control for that form is set in the property inspector.

The property inspector displays only the properties that can be set for the control type. If multiple controls are selected then the property list for all controls is merged into a single list and displayed. If all the controls selected have the same value for a given property then it is displayed like normal in white. If one or more controls has a different value for a given property then that property is displayed in yellow. Checkout the new screenshot showing multiple controls selected on one form.

Last night I got the Form Designer back the same level of functionality that it was before I reworked it to use Form in Form (FiF). This was a big change to make, but it was obviously a necessary change I had to make, not only for the Form Designer, but for the Weld platform in general. During this change I also cleaned up the Weld API a great deal so I'm not doing nasty things like creating direct connections to the database in event code but using the built-in database handles that are defined by the application object. I'm extremely happy with the API at this point. Only a few small issues are left. I'd post some screen shots, but it looks and operates just like the old screen shots from October. Hopefully I will get the little bugs cleaned up enough soon I can do a new screen cast to show how responsive the interface is even when going back and forth to the server.

Basic server side events are now working. I've only impleminted onChange events that run PHP script. I'll probably leave it as is for now as long as I don't end up needing anything else to finish the Form Designer application. Next time I get around to working on server side events, I'll implemint conditional events, then static events and lastly static events with bindings.

I've completely redesigned the session database tables. Instead of having the session information in the "user" table, and therefore a 1:1 relationship, I've created a 1:many table called "user_session". This is an import change I had to make before I released the next version of Weld. Without this change I can't make a demo available on the web site without having a system to dynamically create new users. Since I just want one user called "Demo" I need that user to be able to run multiple sessions at once. Because the session code had already been rewritten into a class several months ago, this was an extremely easy feature to add. I was very very happy with how I abstracted the session code and it really proved it was done correctly when I was able to make a major change to the backend layout of the sessions and it only required modifying the session class and only a couple of lines of it.

However, I did decide that I'm passing way to many parameters around in my class calls. Most of it ends up being information needed to talk to the database. I need to pass this around because of a decision to let the ae_application class actually create the connection to the database. I'm going to change this so that the ae_application class needs an ae_db object passed in on creation instead of all the username, password, server information. The session class can do the same. This will dramatically cleanup the class code and I almost made the change while making the session changes but decided to take one bite at a time.

The other issue that the new session system brings up is how I'm currently resuming sessions. Before I would check the "user" table and if there was a sessionid set I would resume the session when the client connected again. If the client exited cleanly the server would clear this sessionid so the next time the user connected, a new session would be started. Currently it still works this way, but it can't stay that way since this would result in one client being able to hijack other sessions when multiple clients are using the same username/password. What's needed is for the client to write a local lock file with the last sessionid in it. If the client doesn't exit cleanly this lock file will still exist and the client can ask the user if they wish to resume the session. If they do then the client will just send a request to the server to send a full XML update back and if they don't, then the client will request a new session and start fresh.

Refactoring the WeldAE form designer to use FiF is almost done. Most of the effort lately has been focused on writing helper functions to allow two forms embed in a master form find and talk to each other easily. This isn't as easy as it sounds, but I've gotten it to a point that I'm happy with it. The last hurdle to be cleared was how/if a form should be able to trigger an event on another form. I needed this so that when a user clicks on a control in the design view, the property inspector form reloads and shows the properties of that control. I was already stripping all the events and adding a special "onclick" event to each control when the form was loaded for design. However, when this event is triggered, there was no way to make it run a function in the property inspector script. I only had a field for "function_name" in the event definition and would need to add another property to specify the form and script to run. I didn't want to do this because of how messy it would be. For instance, what if the form didn't exist or wasn't currently active?

I decided instead to have the event be client side and just directly populate a hidden text field on the property inspector. This field will have a server side event onchange that will do the heavy lifting of highlighting the selected controls and listing the values in the property panel. Sounds simple, but it caused me to finally run into a piece of the server side code I've been ignoring for a long time. So when the hidden field is changed, a server side event is run. This event populates the current control combo which also has an onchange event that does several things. The problem is that when a property is set server side, there isn't any code to run onchange events. In the set() functions on the Application, Form, Controls and Events, I should be checking to see if there is an onChange event that needs to be run and that code isn't written yet.

For whatever reason, I hadn't run into this yet and I have been quietly ignoring the issue until it caught up with me. This has allowed me to completely flesh out the event model in just one place, client side, without trying to keep the server side in sync. While this has saved me work in the long run, it is a lot of work to sit down and do at one time. I tried to keep events simple in Weld, but they have slowly grown more complex with time. Basic events are very simple and I can just port the C code directly to PHP in less than an hour. Bindings are what complicate matters a lot. I'll probably ignore bindings for now and just get the basic events working and deal with the rest as I run into them. In fact, I'll probably only support script events for now since they're the most basic and should only require a dozen lines of code. Next I'll implement conditional events which should be a straight port of about 300 lines of C to PHP. One day before 1.0 I'll get bindings working server side.

Just got back from a massive on-site job that pretty much took over my life in January. I got a couple of small thinks done on WeldAE, but not much. The good news is that I should have lots of free time in February as compensation for working myself so hard in January. I fleshed out a couple of server side scripts to return the active tab control on a tabpanel. I need to think about sub classing each control type on the server side so I can add specific actions based on the control type. So a tabpanel would have an action called activetab(), but a frame control wouldn't. If I'm not going to have many actions I might just add the actions directly to the control class and test for the control type when run. I'll probably do the later for now since it will be easy to subclass it out later. I also created a function to return an array of controls that have a specific parent. This function would actually be more efficient if added to the control class since it will be shared between several controls that can have children.

All these high level decisions are going to take a back seat to getting the next version out the door and with source code this time. I need to sit down and decide what I want the next release to look like so I can focus my attention toward those goals. There really isn't any reason I can't release today except that there isn't anything nice to look at compared to the last release. There is a HUGE technical difference as WeldAE can actually be used to build real applications now. I've focused my efforts on building a good form designer application within the WeldAE framework. While I still think this is the right direction to go in, this type of application pushes the limits of what WeldAE can achieve and is therefore much more difficult than building a contact manager or something that will demo well.

Despite the title of the last post, I am very happy with where WeldAE is at the beginning of 2006. The PHP backend is in very good shape with only one major change needed to get it where I want it for 1.0. If you had asked me in July where I thought the PHP backend would be on January 1st, 2006 I would have said functional but a mess. Instead, it's elegant, robust and almost in final form.

The Weld client isn't progressing as well as the backend. I've been dealing with resizing issues that just won't seem to go away. I thought I had solved this issue but the recent implementation of Forms in Forms (FiF) has brought this issue back to the surface. Currently there are still issues when using FiF and resizing, but they are to a point where I'm going to ignore it for a bit and move forward with getting something visible done.

In the process of trying to fix the FiF resizing issue, I did accomplish something I'm quite happy with. I completely reworked the splitter control so it is now split into three separate controls. Before it was one control and had a pane1 and pane2 property. This property was set to one and only one control and this control would become the child of that pane of the splitter. This required that all the controls you wanted on that splitter pane be packed into another container widget first. The only container available was the frame control which caused a layout quirks such as a blank spot at the top where the frame puts the label and an extra frame border in each splitter panel. Now that the splitter consists of three controls, a splitter and two splitterpanels, controls can just set their parent to be the splitterpanel they want to be in and it works just like any of the other container controls such as the tabpage and the frame. The two splitterpanel controls are set to have the splitter as their parent with an additional "location" property that determines if the splitterpanel is for the left/right/top/bottom of the splitter. I actually considered this method when I first implemented the splitter control be discarded it as too complex. I was 100% wrong as this new system is much simpler both in the client code as well as for the user creating the control on the backend.

I'm finally done working crazy schedules but with the holidays approaching soon, I don't see getting much more done before the end of the year. Since the last post two weeks ago, I haven't made much real progress. However, I did do some Internationalization work during my day job. The product I was working with did a horrible job and it got me thinking I need to add multi-language capability to WeldAE while it's still in the early stages. Fortunately, since everything is in a simple database this should be easy. The other software I was dealing with used binary blob fields which is why I think their product is not very good at internationalization.

I added two fields to my property table called "language" and "platform". Here is how I envision a simple control would be coded:

Property	Value	Language	Platform
width	50	NULL	gtk
width	60	NULL	win32
text	Hello	NULL	NULL
text	Hola	es	NULL

Hopefully the platform property will almost always be NULL, but I figured I might as well make it available so if there is a platform specific rendering difference it will be easy to work around. If I don't provide this, users will just put platform detection in code if they run into a problem and it is much harder to maintain it there. This way if I do fix a bug I can probably detect the workaround and suggest the developer remove it. At worst it will be easy for the developer to get a report of all the platform specific properties he's using and he can try to reduce them as bugs are fixed.

Finally, I'm still working to knock all the bugs out of the forms inside of forms(FiF) code. I still need to recursively close forms when the parent form is closed and I'm still having some resize event issues. Both of these issues shouldn't be too hard to get going and would probably be fixed already if I wasn't taking the time to cleanup the PHP scripting event API while I'm building the new Form Designer application using the new FiF technique. Most of that work is done and I'm to the point where the Form Designer looks very similar to before, just with less functionality. The good news is that I've already written all the code, I just need to clean it up as I add the functionality back. The new code is much simpler so this is going fast.

My day job went from crazy busy to ludicrously busy. Lots of 8am to 4am days and traveling all week. Fortunately, some of the work has involved massive imports so there is some hurry up and wait bits that I can use to work on Weld in half hour stretches. My most productive session was when bad weather in Atlanta gave me three hours doing nothing but coding. This week should be more normal hours so I'm hoping to get a lot done before I start working like crazy again next week.

I fixed the bad issues with binding properties across forms. This required careful attention to remove binding references as forms, controls, events are destroyed. It took me about 25 hours of debugging to get this figured out because there is a lot of time sensitive issues about how the structure is torn down. The upside is that I'm very comfortable with gdb now and my white board is crammed with very impressive looking memory diagrams.

Now that the plumbing is all done I can focus on getting back to moving forward with the form designer application that will be the bootstrap for all WeldAE development. Now that embedded forms are available I'm breaking the monolithic designer form into pieces. There will be the main window with the menu, toolbar and status bar. The left hand property tab and combo control selector are now a separate form and will use new application level events that fire when controls are clicked/focused to show the properties of controls. This will basically be a generic WeldAE object inspector that should be usable even at runtime for debugging.

Because I started a new form from scratch I had to fix some of the coding uglyness of the WeldAE system. For example, so I had an event that fires when the user clicks a button that creates a control. Previously, here is an example of the code the server ran for the event. For reference, "$this" refers to the script object. The first line actually creates the new control object and assigns it to the form control array. The second line is to dereference the object so it's easier to set all the control properties. The change might seem small, but I can't tell you how many times a typo in one of the three locations the name of the control is needed caused problems. I can also validate the name and type in the "new_control" function before creating the new object.

Before

   function on_click(){
      $this->form->controls["new_control] = new ae_control("new_control", "label");
      $control =& $this->form->controls["new_control";
      $control->set('top', $top);
      $control->set('left', '120');
   }

After

   function on_click(){
      $control &= $this->form->new_control("new_control", "label");
      $control->set('top', $top);
      $control->set('left', '120');
   }

Application level properties are 100% functional now and I've shifted all the special key press properties like "Left Shift Pressed" and "Right Alt Released" to this global property list. Seems like a small change, but now that this application property system is in place, I've got plans to alter the way lots of the WeldAE interactions happen now. Currently the client sends a lot of meta information over when it's trying to authenticate to the server during the first connection using a lot of post fields. The server responds with one of 4 possible responses in plain text. Now I'm thinking I'll go ahead and build a full application struct, put all the information in it and then post the XML to the server. The means a large change in how the initial connection is handled on the server side, but the change should allow anonymous connections that currently aren't possible as well as an easy to add meta data in the future with no code changes to the server. Meta data includes such things as OS, client version, screen resolution, etc.

Application properties exposed...ok re-exposed how bad the PHP XML parsing code was. Instead of trying to hack it up to parse out yet another level, I refactored it and made one small change to my XML layout and now I'm resonably happy with it now. I don't see changing the XML layout again so here is the general outline:

   <xml>
      <forms>
         <properties>
         <event>
         <form>
            <properties>
            <event>
            <control>
               <properties>
               <event>
            </control>
         </form>
      </forms>

Or in a little more detail so you can see how the details work:

   <xml>
      <forms>
         <properties>
            <name>Form Designer</name>
            <...>
         </properties>
         <event>
            <properties>
               <type>onclose</type>
               <...>
            </properties>
         </event>
         <form>
            <properties>
               <name>Property Inspector</name>
               <...>
            </properties>
            <event>
               <properties>
                  <type>onclose</type>
                  <...>
               </properties>
            </event>
            <control>
               <properties>
                  <name>Property Inspector</name>
                  <...>
               </properties>
               <event>
                  <type>onclose</type>
                  <...>
               </event>
            </control>
            <control>
               <...>
            </control>
         </form>
         <form>
            <...>
         </form>
      </forms>

The Forms in form feature is a good example of how a good feature causes a cascade of changes in all parts of an application.

Still very busy with my day job, but I've managed to make progress on some important Weld features. Weld now supports forms inside of forms. This had to be done to move forward on the form designer application since trying to merge multiple forms into one name space was making the code too complex. I don't expect applications to use this feature very often, but it will make navigation interfaces much easier to implement. Think of a navigation system like Outlook uses with icons down the left side that when clicked open forms on the right. Before Weld allowed embedded forms, you had to load the form, copy the controls from that form to the main form, then destroy the loaded form. This worked, and was how forms were loaded for the form designer, but it was less than ideal especially when lots of forms are involved. Now you can just load the form and point the new form to a form and control to embed itself into and forget about it after that.

To embed a form, two properties must be set on the form, parentform and parentcontrol. Currently you can only embed a form into a control and the control must be a container such as a frame or tabpage. However, it will be able to be place directly on another form as soon as all the other embedding issues are resolved.

There is only one real issue and it should be easy to fix once I figure out what's happening. It has something to do with binding properties to properties on other forms. When a form with bindings is closed, the property on another form that is bound to it has issues. At least this is the theory. I've written cleanup code to find these bound properties and remove them, but something still isn't working like it's supposed to. Hopefully I can track the problem down over Thanksgiving and move forward with a new release in December.

I've been just too busy with my day job to work on Weld for long lengths of time. Because of this, I've been working on small cleanup tasks that I've been putting off for one reason or another. Hopefully I'll have more time in November and December to do a couple of the big ticket features I want to add.

The Weld GTK client now compiles cleanly with GCC4. It has always compiled and run fine with GCC4, but there were dozens of warnings about char pointers not being the same sign as the target. Apparently GCC4 is very picky about this. LibXML2 uses signed char pointers and everything else in Weld just uses just char pointers. Fortunately, this bit of the code is separated very cleanly and it wasn't too bad to cleanup. Once I figured out how to cast from a literal to something LibXML2 could use and from a LibXML2 xmlChar pointer to something strcmp() could use it only took about 2 hours to clean it all up. I probably researched the problem for another 3 hours.

PHP5 has been a much harder problem to solve. WeldAE pushes the envelope of what PHP can do and some parts of the code did things that PHP5 didn't find funny. I had actually gotten WeldAE working on a beta of PHP5 a year ago, but I had to do so many large changes to the code, it would no longer run in PHP4 any more. I scrapped a weeks worth of work and went back to the PHP4 code base. Using what I learned during the PHP5 port, I slowly made the PHP4 code base more PHP5 friendly each time I cleaned up a section. When I got my new laptop and installed Fedora Core4 which comes with PHP5.1 I decided it was time to give it another go.

Turns out all my cleanups over the months put me in good shape to get WeldAE working with version 5. I had to quit using vars name $this which was legal in 4 if you weren't inside a class but in 5 is illegal anywhere. The biggest problem is that the current index pointer for arrays works a bit different in 5 than in 4. In 4 the current index is the first index unless you move it. In 5 it seems that the current index is the last index populated or something along those lines. It took me a long time to realize what the problem was, but once I found the cause, I simply had to make sure I was calling reset() before looping through and array with list() or foreach(). All in all, PHP5 has increased the quality of the sever side PHP code.

As I mentioned last time, I'm traveling a lot currently which has impacted the amount of time I can spend working on Weld. However, I have managed to put in a few hours and since I'm traveling I figured it would be a perfect time to take a look at why my Form Designer application starts so slow when running it over the Internet as opposed to a LAN connection.

I'm staying at a hotel with "High Speed Wireless Internet access", which means I get ~12k/sec download speeds and 100ms ping times. I guess they have a different definition of "High Speed" than I do. This should be about as bad a connection as most users are likely to see in the wild unless you have dial-up Internet access. I'm not really targeting the dial-up crowd, so this is a good "worst case" test.

First, I wanted to get a good initial benchmark before I made any changes. I put an "exit(0);" directly after the form render code so when I launched the client it would download the user's default form, render it and then exit. I also dumped the RAW xml to stdout so I could see how much data was being sent. I did this on my old 433mhz laptop, so times on a faster machine should be better by 0.5 second or so. Here are the results of the inital benchmark:

Time: 22 seconds
Size: 180K

Wow! I knew it was taking about 20 seconds to start, but I didn't realize it was sending 180K of XML. After looking at the XML I realized that I was sending over all event properties even when they didn't contain any data. This is a big deal for the Form Designer application since it uses a massive amount of events most of which should just have one property, the server side script to run. I made the event XML generation work just like the control and form generation and just send properties that actually have data. I had done a lot of event cleanup on the client a month ago that made this process simple. Before the client cleanup, I HAD to send all these fields or the logic code wouldn't work. I guess I fixed the client and then never fixed the server. This got the XML size down to 100K

Next, I was able to claim a few more bytes by not sending blank error tags for controls. I also removed a bunch of white space, but not all of it. I left the white space after closing Form, Control and Event tags for readability. Eventually I'll remove all the white space when I actually start pushing the XML into a tree widget for debugging. Until then, I debug using the RAW XML, so I have to keep it somewheat readable. Here is where I now stood:

Time: 9 seconds
Size: 80K

The XML is now clean as it can get without using shorter tag names, which I'm not going to do. The only other way to reduce the size is to compress the data using gzip. I added an output handler to my sever side PHP code and verified that the server was compressing the XML, which it was. This reduced the data sent to 5k! However, I was still seeing 9 second render times on the client, so libcurl either wasn't requesting compressed data or couldn't handle compressed requests. After some reading, I found that by default libcurl doesn't request gzip compressed data. After adding the "request gzip" option to my curl code, here is where the benchmark ended up:

Time: 1.3 seconds
Size: 5K

Needless to say this was a better increase than I expected. Weld now runs almost as fast over a slow Internet connection as it does on my local LAN. I can still tell the difference, but it is only noticable when starting up the Form Designer. Dialog screens are instant because they are so simple and these account for most of the interaction with the types of applications Weld will be used for. This has got me fired up to release a new demo version of Weld as soon as possible.

I've been traveling a lot recently which has really cut into the amount of time I can work on Weld. Luckily I got a lot done at the beginning of the month before my schedule got too crazy. Here is the list from the last post with everything completed marked out and additional fixes and comments in green. I know I've left out 90% of the changes I've made over the last month, but these should be the highlights. I need to get a changelog process in place to make these updates more comprehensive.

When selecting multiple controls, I want to dim the control picker text to indicate to the user that this is the last selected control, but there are multiple controls selected.
When selecting multiple controls, I currently only display the properties for the last selected control. I need to instead all unique properties for all selected controls.
When selecting multiple controls, I want to color code the properties when:
- This property is set to the same value on all selected controls
- This property is not the same value on all the controls
- This property is read-only on some controls selected. (Calculated value for instance)
Need to flesh out the tab panel control with properties for active tab and tab change events.
Image cache This is done to a point that it's good enough for now. Currently the client ignores the image date sent by the server. It just uses the local file if it exists and downloads the file if it doesn't. Shouldn't be hard to fix since all the hard work is done, but it isn't a priority right now.
PHP structure cleanup ("form->get('name')" instead of "form->name" for instance).
Need a much cleaner API for adding events, controls and forms in PHP.
Need to add true control selection with grab handles for resizing and not just changing the background color of the control The day this gets marked off my todo list will be a good day. I'm not 100% sure I can do this on either Win32 or GTK, much less both. I read something on the GTK mailinglist during the week that I want to try out, but don't expect this anytime soon.
Get the global application properties syncing between the client and the server and move the meta key status from the form to the application property level
Fix form and tab panel resize issues where scrollbars are sometimes displayed even though they aren't needed. This has been a BIG issue for a long time and I thought I would never completely fix it. I've been working on it off and on for months and I've finally completely fixed the issue to my satisfaction! When I get back to working on the Windows client, I'll have to go through these battles all over again and probably worse. GTK is VERY good about reporting correct sizes of everything. Win32 is horrible from the bit of Weld I've done so far and I'm sure it will be a struggle. Fortunatly Win32 is easier in some respects because it doesn't have themes and 40 different window managers.
PHP structure cleanup for user scripts. I didn't include this in the above structure cleanup since it's much more involved. All scripts now inherit from ae_class_script.inc.php which can be used to create vars and functions useful when your running Weld events. No functions yet, but I created now instead of each function taking pointers to the current form and control, you can just use the class vars $this->form and $this->control. $this->db and other "global" vars are coming soon.
Added a system for Weld to deal with files. This system includes a table to store information about the file such as with application it belongs to, where it's located, etc. It also includes helper functions for generating URLs that point to a given file. Just give the function the name of the file and which application is belongs to and you get back a full URL pointing to the file as well as the current modify date of the file. The entire server side system is pretty complete and is the hard part of the image caching system mentioned above.

New Items that need to be done:

Weld Core:

Ability to put background images on tab pages. I need this so I can put a grid on the form tab in the form designer application. Should have been easy, but I'm having no luck using the same code I used to put images on form backgrounds.
Need to reword grids and combo box data binding. Binding data controls and column controls to grids and combobox controls is very brittle and depends on the server sending the controls over in the correct order to work correctly. If a data control is bound to a grid at runtime, this breaks te form designers ability to select the grid since a grid without data doesn't receive events in GTK. I need to have a global fake data control to bind the grid to until it's real data shows up.
Might need to have a toolbar control. Currently I'm faking one with image buttons, but Win32 doesn't support buttons without borders so it doesn't look like a real toolbar. I'll probably stick with the fake one for a bit as a place holder, but the writing is on the wall for adding control(s) to Weld for toolbars.

Form Designer

Add the ability to add new controls to a form. I've got the toolbox tab populated with icons of each control type, I just need to add the interface to picking and placing the control on the form.
Need the ability to save the form back to the database. This involves a server script that will walk the structure and insert/update the database. It shouldn't be too hard to do a first draft of this script, but I could spend a year optimizing this routine to use defaults to limit the data size of each form. There is also versioning to be thought of, which needs to be thought through carefully.
Need to add the ability to "test" a form while designing it. I haven't decided if I'm going to require that the form be saed back to the DB first or if I'll write the current form to a temp location and then run this temp form.
Copy/Paste control support.
Need to display the form properties when clicking on the form. Probably shouldn't let is be selected along with other controls at the same time.

Despite hurricane Katrina leaving me without power most of Monday and Tuesday, I managed to make amazing progress on the Weld client and the form designer application. Lots of small fixes to the memo, combobox and splitter controls as I pushed the limits of these controls with the new development on the form designer. I open the designer with four full forms in design mode to stress the load speed and make me aware of any slowness I might introduce during development. So far it's very fast with almost no delay even though I'm constantly starting/stopping it to test out new code.

Some of the large changes include the ability to select controls on the form designer and have them appear selected. The selection system also supports multiple controls selected at once by holding the Ctrl key down while selecting a control. To show which control is currently selected I added a combobox control picker above the control property list. When a control is selected on the form designer, the name of the control is displayed in this field. If a control is picked from the drop down then the control is then highlighted on the form designer. As a bonus, if you hold the Ctrl key down and select a control from the control combobox, it does what you would expect and highlights the new control in addition to the ones already selected.

There are now two splitter controls on the main screen allowing the user to control both how wide the left property panel is but also how much space is used by the new property description field. Currently the description is pretty slim, but I hope to one day have complete and lengthy formatted descriptions. My goal is to limit the need to open the documentation just to figure out how a property/control works. Some other small changes are mock-ups of more menus/tabs and a new toolbar section.

The biggest crunch currently is that I want to add a bunch of images to my interface but I don't have any image caching in place yet. This severely impacts startup speed since each image has to be downloaded at startup from a remote server each time even though they rarely change. There are so many ways to implement an image cache and even more ways to completely mess it up, I'm a little worried about this bit. I can't put it off much longer so I'll probably use whatever method is quickest and think on it more before putting the time in it required to do it right.

There is still a lot of little things to be done:

When selecting multiple controls, I want to dim the control picker text to indicate to the user that this is the last selected control, but there are multiple controls selected.
When selecting multiple controls, I currently only display the properties for the last selected control. I need to instead all unique properties for all selected controls.
When selecting multiple controls, I want to color code the properties when:
- This property is set to the same value on all selected controls
- This property is not the same value on all the controls
- This property is read-only on some controls selected. (Calculated value for instance)
Need to flesh out the tab panel control with properties for active tab and tab change events.
Image cache
PHP structure cleanup ("form->get('name')" instead of "form->name" for instance).
Need a much cleaner API for adding events, controls and forms in PHP.
Need to add true control selection with grab handles for resizing and not just changing the background color of the control
Get the global application properties syncing between the client and the server and move the meta key status from the form to the application property level

It's been a while since I last posted an update on Weld progress. I'm happy to report that despite July/August being a very busy month for me with other things, I've made solid progress.

With help, I'm well on the way to having a complete expression system for the binding system. It's a horrible bit of code currently but it works and we should be cleaning it up shortly so it can be turned on for use.
Completed the huge PHP refactor.

Added properties to the application boject for storing of global information.
Each form now has a master script that can have multiple functions in it. These functions are wrapped as a class to provide namespacing. This will dramatically simplify forms that have lots of code behind them since all the code can be located in one script.
Removed all recursive object references which isn't supported by PHP's serialization methods. Instead of linking to say a controls parent form in the object, I added a class method that walks the object and returns the controls parent. I can always add caching later if this proves to be slow.
Fixed all the string escaping issues

Fixed a big issue where server side events were firing while the form was being rendered. This resulted in partially initialized controls. Now I disable server side events until the form has been rendered. I might disable all events in the future while rendering, but a lot of events need to fire such as form/control resize events.

Fixed all the name spacing issues in the form designer so multiple forms can safely be loaded without stepping on each other. Finished the code to handle loading the form, disable events, fix the namespacing, etc. I also check to see if the form is already loaded. If so I need to switch the active tab to that form, but currently that's not supported by the Weld tabpanel control. That will be fixed shortly.

Created a new "designmode" property for all controls that changes how the control is rendered when it's set to TRUE. I'm having problems implementing this for all controls because of how events work in GTK, but I'm sure I get it figured out soon. The textedit controls works perfectly, allowing you to click on the control to get the properties but not allowing you to edit the text directly. Eventually I need to figure out how to put a bounding box around the control for sizing and moving.

The big noticeable difference since in the Weld client since my last post is the new splitter control. This control lets you place controls in two containers with a handle in the middle that is used to resize the containers. It's a fairly advanced control that I don't see being used too often but I really needed for the form designer. With this new control it is possible to control the width of the property panel/toolbar which really helps it feel like a true application and not just a toy.

Other than the one new control I've spent quite a bit of time refactoring the PHP side of things. I've decided to require a DB on the server side which means I can make the code less abstract and more straight forward. I failed to use class extensions for adding DB support and had to go back to just including functions into the global name space. It's still clean the way I'm doing it but I've got to eventually find a better way for this when I add support for other DB systems.

I was successful in adding a base "application" class to the PHP side. When I first wrote the PHP side three years ago I just structured it as an array of forms. By the time I was done with the first prototype and writing the first real client I realized I needed a base class that could hold session information, including application wide global vars. Getting this structure setup on the PHP side is the first step and I should be able to add a property array for user defined global vars soon. Basically this would allow a developer to store information that could persist for the duration of the application. This allows for easy passing of information between forms even if the source form is no longer open.

Finally with a lot of help, we're well on our way to being able to use arbitrary expressions in binding paths. Currently the only two operators supported are addition and subtraction. When we're done binding paths will support:

Addition [+]
Subtraction [-]
Multiplication [*]
Division [/]
String Concatenation [.]
Order of Operation [()]

A typical binding currently looks like {PARENT.width} - {.left} - {.left} - 10. When we get the new system in place it will be possible to do more complex bindings such as {PARENT.width} - 2 * ({.left} - 10).

It's been a very busy month since my last post. Typical summer events have limited my ability to work on Weld and I haven't made much visible progress since the last post. However, I have solved the two big problems that have been longtime issues. I have eliminated an obscure memory segfault which was holding up my progress on the form editor. I also fixed a strange onChange interaction with text edit controls that was causing the text property to blank out a lot. I spent about 20 hours actively debugging these two problems and untold time thinking about the issue.

Having these two issues behind me should mean rapid progress on developing the first Weld application, the form editor. As long as I don't run into bugs in the client this should progress quickly. However, since this is probably the most advanced application that will ever be coded in Weld, there is a good chance I'll run into at least some issues as I push the boundaries of what the Weld client was designed to do.

My plan had previously been cleanup of the PHP back end. However, now that I've solved the hard client side issues, I'm more likely to push forward on the form editor. This will give me something impressive to show off the progress made since 0.1. Cleanup on the PHP back end has to be done eventually but I want to work on something more user visible.