Home » MODDING HQ 1.13 » v1.13 Idea Incubation Lab » xml overhaul
| xml overhaul[message #325849]
|
Sat, 28 September 2013 17:00 
|
|
 grodrigues |
 |
Messages:27
Registered:June 2010 |
|
|
Intro.
Most jagged alliance 2 data files are xml files, but xml editing is, quite frankly, a mess. Bugs in the xml editor are frequent (e.g. stripping tags) and modders, both newbie and veteran, seem to resort to hand-editing, which is inefficient and error-prone. Given that a large part of mod content is xml, especially in item mods, the situation is far from ideal.
This is a proposal, not just to the core developers but to the community at large, to improve the situation. Being a proposal, I am of course open to suggestions for improvement, but certain key ideas are not open for negotiation and their rejection is, as far as I am concerned, tantamount to a rejection of the proposal as a whole. But even if it is rejected, the work already done is offered to the community if it wants to make any use of it.
To see where I am coming from, some caveats are in order:
(1) I play jagged alliance 2 v1.13 (ja2 for short) on kubuntu via wine.
(2) I only use the xml editor to try it out; if I want to edit xml I just open a text editor.
(3) Although I am proficient in C++, the language is notoriously brittle and difficult to master, and I have little to no desire of wading into the ja2 codebase. And even if I had, the fact that I am on linux poses some substantial hurdles to compiling ja2.
I will divide my musings in several posts for ease of perusal.
Report message to a moderator
|
Private 1st Class
|
|
|
|
| Re: xml overhaul[message #325850]
|
Sat, 28 September 2013 17:01 
|
|
| grodrigues |
|
Messages:27
Registered:June 2010 |
|
|
Desiderata.
Before any solution is considered, I will list what I think are its desiderata.
(1) compatibility: if the game receives an xml in an unrecognizable format, it barfs and exits.
As I said previously, I do not know the ja2 codebase and have little intention of knowing it, so any proposed change is a load on the developers. So, in order to have even a modest chance at acceptance, the load must be reduced to a minimum. In particular, anything more than plain, simple xml (e.g. namespaces) is a no-no.
(2) modder friendly: if it does not make life easier for modders, who are not expected to know xml or programming, ditch it. Simple.
(3) fault tolerant: as far as possible, errors should be caught and signaled early, but where some xml fragment is not recognized, it should be left alone.
This bullet actually contradicts (1), but if such bugs as the stripping of tags are to be avoided, this is plainly necessary. Furthermore:
(4) future proof: as far as possible, it should leave room for future extensions.
(5) platform independence: ja2 is for windows but I play on linux, so this is a convenience, but it is also pretty much forced upon me.
Report message to a moderator
|
Private 1st Class
|
|
|
|
| Re: xml overhaul[message #325851]
|
Sat, 28 September 2013 17:01
|
|
| grodrigues |
|
Messages:27
Registered:June 2010 |
|
|
Solution.
I propose to satisfy the desiderata by a mix of two solutions:
(a) An external library providing a programmatic API to deal with xml, and tools using the library (see below for what I have in mind).
(b) A proposed standard format for ja2 xml files. Because of (1) of the previous post, this standard can only be a *suggestion*, but on the other hand, having a standardized format is crucial for the whole project to work. Because of (a) above, the standard can be made into a *workable* suggestion: transform from ja2 original xml into the desired format, apply needed transformations, transform back to ja2 format.
As of this writing, the following has been done:
(1) a draft for an xml standard format of ja2 data files is written.
(2) a library is written in Python. It consists of essentially two layers: a Pythonic layer around the the xml DOM level 2 compliant API, and then a library on top of this layer to deal with the ja2 data formats proposed in the standard mentioned in (1), including some common data transformations, data queries, etc.
Report message to a moderator
|
Private 1st Class
|
|
|
|
| Re: xml overhaul[message #325852]
|
Sat, 28 September 2013 17:04
|
|
| grodrigues |
|
Messages:27
Registered:June 2010 |
|
|
Test case.
Instead of explaining the above two bullets in any detail, it is better instead to work out a test case. Consider the file facilities.xml, the file that controls the placement of facilities in the strategic map. Whip up a script to transform it into facilities.new.xml:
FacilityType
/home/goncalo/documents/games/ja2/modding/mods/xml-overhaul/src/scripts/data/facilitytypes.new.xml
szFacilityShortName
Facility short name in FacilityTypes.xml.
FacilityPopup
always
0
explored
1
never
2
Symbolic names for facility popup status info.
Facility
coordinates
String
Sector coordinates.
facilitytype
FacilityType
Facility type as in FacilityTypes.xml.
hidden
FacilityPopup
Whether the facility appears in the strategic info popup.
Sector facility.
B13
Airport
always
The key points:
(1) Absolutely *no* hand-editing was done to transform the original facilities.xml into facilities.new.xml. Everything was achieved via Python scripting.
(2) The xml produced is valid, human-readable and prettyfied; in particular, it is hand-editing friendly.
(3) The format is self-describable: not only the types of the various fields are specifically declared, but human-readable descriptions are also included. This is *the crucial* feature of the proposed xml format, as it is what allows everything else.
note: the example above is actually not fully conformant to the standard I have drafted, but the difference is small and easily patcheable.
(4) The xml was also annotated with metadata: in this case, the attributes of the [table] root tag specify the author, the ja2 revision (needed to synch evolving formats) and the owning mod.
(5) The structure of the xml is rigid to a certain extent: the code expects to see certain specific elements at certain specific places. But otherwise, anything not recognized is left alone. This is seen in FacilityTypes.xml (not showed) where the declaration of the main record type is "partial", defining only the absolutely necessary fields.
(6) An external xml file is referenced; I have inserted an absolute path for simplicity, but this of course is a no-no. The paths should be instead relative to the *mod* that owns the xml file, or at least relative to the ja2 executable. I have opted for this quick and dirty hack, because the logic to get the full path from the mod-relative path still needs some ironing out.
(7) Some extensive renaming was done; of fields, their types, etc. This was more to showcase the capabilities than a real need, although in all honesty, the naming conventions used in ja2 are just dreadful. Hungarian notation? Seriously?
( A more substantial change, and this time a useful one, was the inclusion of an Enum type to document, and programatically constrain, the possible values of the field. Similar validators for other fields (e.g. a field whose value must be within a certain range) are possible to include, but *how* precisely to do that depends on the standardized xml format.
(9) Another substantial change was making the xml reference the szShortName field instead of the ubIndex field. Once again, this was done to showcase what can be accomplished *now*. And it *is* useful for hand-editing (although a proper xml editor bypasses this need).
A very simple logging framework yielded the following:
Read xml file /home/goncalo/documents/games/ja2/modding/mods/xml-overhaul/src/scripts/data/facilitytypes.xml.
Normalizes xmltree to valid Table xmltree object.
Replace top node in [data] with its children.
Analyze first node in [data] and append Record declaration with same skeleton structure.
Rename Record declaration.
Write xmltree Document to file /home/goncalo/documents/games/ja2/modding/mods/xml-overhaul/src/scripts/data/facilitytypes.new.xml.
Elapsed time: 0.022 seconds.
Read xml file /home/goncalo/documents/games/ja2/modding/mods/xml-overhaul/src/scripts/data/facilities.xml.
Normalizes xmltree to valid Table xmltree object.
Replace top node in [data] with its children.
Analyze first node in [data] and append Record declaration with same skeleton structure.
Add descriptions to Record declaration and its fields.
Rename Record declaration and field records.
Prepend Enum declaration for facility popup status.
Remove root comments.
Change the type of field hidden from Integer to Enum.
Prepend Reference declaration for FacilityType field.
Change the type of field facilitytype from Integer to the FacilityType Reference.
Change the field referenced by type FacilityType of field facilitytype from ubIndex to szFacilityShortName.
Delete all facilities except the first one.
Delete all comments except the first one.
Write xmltree Document to file /home/goncalo/documents/games/ja2/modding/mods/xml-overhaul/src/scripts/data/facilities.new.xml.
Elapsed time: 0.035 seconds.
Timings were included because at first, I was afraid that the code would be slow. Python is a scripting language, the pythonic layer is such that seemingly simple operations can trigger a traversal of the whole DOM tree and furthermore, the code itself is fairly straightforward, with hardly any optimizations in place. The second file processed took about 50% more, but there is significant more processing being done, including the opening and processing of a second xml file. So, for now, performance seems ok, which is all that is needed, although some stress testing is planned (e.g. processing items.xml).
Report message to a moderator
|
Private 1st Class
|
|
|
|
| Re: xml overhaul[message #325853]
|
Sat, 28 September 2013 17:05
|
|
| grodrigues |
|
Messages:27
Registered:June 2010 |
|
|
Pros.
Here are some of the selling points of the proposed solution.
(a) An interface to deal with xml data programatically: this allows to satisfy (1) as already delineated, e.g. transform from ja2 original xml into some other format, apply needed transformations, transform back to ja2 format.
(2) It is relatively easy (it was coded with this feature in mind) to do the following: a gui xml editor. It takes off some of the load of the developers, and provides a good entry point for modders, newbie or not, to do xml monkeying. Being written in Python, it is much easier to change it, improve it, fix it, etc.
This is a general point worth elaborating on: the *only* thing that should be in the ja2 codebase is the absolute necessary to run the game, say the code for reading the xml data files and populating the internal C++-objects. Everything else should be delegated to external tools to alleviate the codebase and free the programmers for more useful things, or to put it in another way, if the work *can* be delegated to an easily available external tool, it *should* be so delegated.
(3) It is relatively easy (it was coded with this feature in mind) to fully convert from and to sqlite (*) databases. Not only with the structure but populated with data read off from the xml. And back. This provides another solution to the xml editor as there are several, freely available tools to edit sqlite databases.
(*) Or any sql database engine with Python bindings, that is: pretty much every single one under the sun. Sqlite is singled out because it is lightweight, needing no complicated setup or a separate server process, and the Python standard install comes with bindings for it.
(4) Modder-friendly in the following sense: whenever a change in the structure or content of the xml is needed, it is easy to bundle a script to transform the old xml into the new xml. This deals away with one class of errors. The modder then only has to fill in the data needed, which can be done the old fashioned way, or once again, via scripting.
Q: Are you claiming that it is possible, with the use of this library, to automatically upgrade an xml file targeted for gamedir revision x to gamedir revision y > x?
A: Almost. The library does not automatically put in new values, outside of some chosen default value (that needs programming skills still not available: mind reading), but everything that does not involve getting such new data can be accomplished: renaming or moving files around, xml restructuring, type conversions, etc.
(5) Extendability: xml is a data format. It is relatively easy to replace the ini files with xml configuration files -- the draft format already has a proposed standard for these config xml files, and some support in the library is already in. Furthermore, it is easy not only to read the config xml files that are used by the .ini editors but also the .ini files themselves. Less work for the core developers, as the ini editors would no longer be needed.
(6) Future friendly: besides the suggestions adumbrated above, I also have some ideas to replace the current virtual file system (which imho, is badly designed) with a full mod slash plugin system, driven, once again, by xml files.
Report message to a moderator
|
Private 1st Class
|
|
|
|
| Re: xml overhaul[message #325854]
|
Sat, 28 September 2013 17:06
|
|
| grodrigues |
|
Messages:27
Registered:June 2010 |
|
|
Cons.
Here are some cons. These are the ones I came up with, but by all means add your own.
Objection 1: everything that can be accomplished by this tool, can already be done on the C++ side, so why bother?
Answer: possibility is not actuality. Sure, a C++-proficient coder could do it, but why should the developers, whose job is after all the maintenance and improvement of the ja2 codebase, be so burdened? On the other, text processing, of which xml processing is an example, is the typical job for which a language like Python is well suited *and* for which an external tool is preferable to any in-house coded tool.
Objection 2: an external tool, with all its dependencies, is needed.
Answer: true, but I think this is a pro not a con (the pun is irresistible), for the reasons stated in (2) of the previous post. On the other hand, one has to consider that the feature is optional, so users do not pay a price in having it. Besides, the external tool needed is a Python installation, which is freely available and easily installable. Furthermore, it is even possible to ship the code and the Python binary, saving users the trouble of downloading Python and installing; however, given the nature of the tool, I advise against this solution.
Objection 3: an extra programming language is needed, Python. The codebase is in C++ with Lua scripting.
Answer: true, and this objection has real bite. However, how many of the core developers know Lua? Personally, I know C++, but C++ is horrible for the kind of processing needed. I never coded anything in Lua, while I am proficient in Python. More importantly, the standard Python install comes with batteries included: xml DOM and SAX interfaces, a gui toolkit, a sqlite interface and a parser for ini files. Besides, the code is open and available for inspection and, imho, clean, well-commented, straightforward and simple, with no deep black magic (e.g. metaclass wizardry).
Objection 4: since it is a programmatic library, and in Python, is there even anyone besides the author that can take advantage of it? And if there is no advantage to anyone, it is of no use, and what is of no use can be discarded.
Answer: true, but it opens up a path where modder-usable tools are easy to come by. See the points in the previous post.
Objection 5: seems a lot of work for little gain.
Answer: lot of work, I agree, little gain do not; it is why I bothered in the first place.
Objection 6: big ambitious projects are dime a dozen. Say, the core developers are generous and believe the author will deliver something usable. Say the core developers accept the proposal; then the author disappears (gets hit by a bus, looses interest, whatever); then who will be left to take care of the code?
Answer: not much I can respond to this. It is a risk with every code accepted in an open source project. If it is any consolation, Python is easy to learn, so if anyone interested is up for the challenge, I envision no major hurdles.
Report message to a moderator
|
Private 1st Class
|
|
|
|
| Re: xml overhaul[message #325855]
|
Sat, 28 September 2013 17:07
|
|
| grodrigues |
|
Messages:27
Registered:June 2010 |
|
|
Current state and TBD.
The basic infrastructure is layed out. It was coded in a 2-3 weeks, in the spare time. What is still to be done:
(1) unit testing: basic tests are in and the worked out example is a test in itself, but definitely some more stress testing is needed to weed out bugs as, for example, the DOM API has some pitfalls (e.g. do not mutate the DOM tree while traversing it). Expected difficulty: easy, but some work.
(2) the library itself needs some extension. For example, the code to analyze a given xml node and return a skeleton of a Record declaration does not recurse to analyze children.
(3) framework extension: in order to increase the usability of the library and get the tools up and running, some more infrastructure is needed: code the logic for getting the absolute path from a mod-relative path; a more complete logging framework; a scripting framework to handle gamedir revisions, etc. Expected difficulty: easy but once again, some work. The problem here is not so much what there is to do, but what I need to do but still do not know that I need to do it. Heh.
(4) the gui layer and the xml editor: this is the minimum of minimums to justify the inclusion and use of this tool. Expected difficulty: moderate. Moderate, because I have to learn the tkinter toolkit that ships with Python, but otherwise, the logic of the code is fairly straightforward (assuming the standardized xml format, of course).
Once this point is reached, the library is ready for consumption, and we can go on to stage 2.
(1) xml file revision: this can be done in a piecemeal fashion, as developer and community help input is needed to go through all the xml files. Expected difficulty: easy, but a lot of work.
Once this is done, we can start looking at the future.
(1) ini files: ini files can be automatically converted to (standardized) xml config files, and thus edited by the same xml editor. The tools are already in place, but there are some details to solve (e.g. how to deal with different languages). Expected difficulty: easy.
(2) plugin xml format and API: as I said above, the current virtual file system for mods and plugins is badly designed imho. With the conversion of ini files to xml made possible in (1) there is further incentive to re-design a new xml-centered mod / plugin system. I already have some ideas for such a framework, but more work is needed. Expected difficulty: unknown.
And while we are at it:
(3) the sqlite layer: if one can easily build a gui tool, programatically driven by the xml spec, the gui layer is right at the corner. Expected difficulty: easy.
The pie in the sky idea:
(4) embed Python in jagged alliance: this would allow the ja2 executable to call Python code, and in particular, to call directly the library and defer to it xml processing or read from sqlite databases as in (3) above. Unfortunately, there are some serious hurdles. Embedding Python is not an easy task (partial answer: boost.python can ease the pain somewhat); I would have to wade in the ja2 codebase to code the Python - C++ bridge; even if I were willing to do this, I am working on linux, so presumably, I would have to get a windows compiler and needed libraries and run it through wine to be able to compile the ja2 source, which carries with it its own sleuth of problems. And most probably even more troubles that I cannot fathom at the moment.
Report message to a moderator
|
Private 1st Class
|
|
|
|
|
|
|
|
|
|
| Re: xml overhaul[message #325860]
|
Sat, 28 September 2013 18:35
|
|
| grodrigues |
|
Messages:27
Registered:June 2010 |
|
|
pheloncabmy question is, how does your tool help this happen? I get where it tried to standardize the xml.. but will that keep the xml editor updated? or does it just clean the xml up to make it pretty yet not actually fix the issue (devs work on the new stuff and not the editor).
There are different questions here, so let me try to answer by considering the following scenario: suppose the proposed xml standard is accepted and the xml editor and the tools that allow it, as I envision it, are in. Now, the following happens:
(1) Because the xml format is self-describable, the logic needed to edit the xml is already in the xml *itself*. The editor only needs to understand it and do the Right Thing. It is the data that drives the code, not the other way around.
(2) Now assume, that there is mismatch in what is declared in the metadata and what is in the data. The mismatch can have two sources: a genuine error or some addition. Either way, the editor should *not* mess the xml in any way, what it should do instead is flag the mismatch and present the data as plain xml and let the user figure it out. In fact, most of these mismatches can be caught by the tool itself, so that the task of validating the xml can also be automated.
(3) Suppose it is an addition, that is, suppose developer X comes along and wishes to introduce this Super Cool New feature. Suppose his changes are committed in gamedir revision y. As long as the changes needed consist in things like adding new tags, changing types, etc. the editor will *still* do the Right Thing with *no* changes needed, once again, because the xml follows the standard and is self-describable, or to put it in other words, the only thing the developer needs to do is, on the xml side, to edit the xml itself to record the changes needed.
The tools also help the developer (and modder) in the task itself. For example, suppose the Super Cool New feature needs the addition of a new tag to some xml node with type Integer within a certain range and a certain default value. Being a C++ wizard, he has no problem in changing what needs to be changed in the executable side. On the xml side, a couple of lines of Python code and voila, everything is updated with sensible values -- and of course this type of mundane tasks should be doable in the editor itself, so no need to learn Python. Furthermore, the tools give the possibility to automatically update the files produced by modders prior gamedir revision y to gamedir revision y by shipping with ja2 1-13 a script to do it. As long as the modder has Python installed, running the script is all that is needed to update the xml files on his side. The update is limited in what it can do (it cannot guess new values, after all), but in the overwhelming majority of cases it is sufficient to upgrade the xml to a format that the executable accepts without barfing. Then the modder can open the editor and add his own mod-suitable values.
(4) Now suppose developer Y is even more ambitious and thinks the xml standard *itself* should be changed in some way. That of course needs an update to the code, either in the editor or in the supporting tools; how serious depends on the change.
Report message to a moderator
|
Private 1st Class
|
|
|
|
|
|
|
|
| Re: xml overhaul[message #325894]
|
Sun, 29 September 2013 03:38
|
|
| Faalagorn |
 |
Messages:154
Registered:February 2012
Location: Poland |
|
|
From what I got, the original approach was supposed to, among all, make the XMLs easier to edit by hand. However, with my very limited knowledge to XMLs and no real coding knowledge, thew new Facility.xml file was simply harder to read for me, mostly because of the whole new lot of metadata I don't know how exactly work and what are supposed to do (I assume that they are here to be able to output something other than numbers.
More so, given the ability to enter text instead of numbers doesn't seems to help me at all. Actually, it's way more confusing for me - I just open the file, read the comment at the top and write the number, while with the text I do have to worry about the proper formatting, typo, etc - and my god if I every will have to work with non-english descriptions, having trouble inputting character and sometimes even displaying them correctly, just to make the entry working.
However, I do agree that one standard of the XMLs is a good thing that will benefit 1.13, and any cleanup/easier readability is much welcome.
I also like Moa's idea of making the XMLs to work under any browser and I don't see an issue with descriptions placed as a field for the first entry (though I have to admit, I prefer them being a comment as far as readability goes. But I understand that using comments as description may bring some troubles later on)
pheloncabWell then, as long as you get the xml editor working with whatever changes you do, and its clearly documented on a good how to for those of us who don't code, go for it.
As I understand it, this is part of what the branches in the SVN are for.. so you can make it and test it etc and it won't be commited back to the main trunk until its working and accepted.
From what I understood (but I may be wrong), this idea have nothing to do with the XML editor. It will just change how the XMLs looks, making them all follow one standard and be easier to edit manually. It will neither fix or break XML editor's ability ot read and edit files. Moe's idea, however, may be a solution to XML editor being out of date, even making it obsolete in the end.
Report message to a moderator
|
Staff Sergeant
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: xml overhaul[message #325967]
|
Mon, 30 September 2013 14:48
|
|
| grodrigues |
|
Messages:27
Registered:June 2010 |
|
|
FaalagornFrom what I got, the original approach was supposed to, among all, make the XMLs easier to edit by hand. However, with my very limited knowledge to XMLs and no real coding knowledge, thew new Facility.xml file was simply harder to read for me, mostly because of the whole new lot of metadata I don't know how exactly work and what are supposed to do
As a user, the metadata part is precisely the part you do not need (or should not need) to look at. Even if you are editing the xml by hand, with a text editor. The metadata is there to make the xml self-describable and to allow the editor to do the Right Thing. In particular:
FaalagornMore so, given the ability to enter text instead of numbers doesn't seems to help me at all. Actually, it's way more confusing for me - I just open the file, read the comment at the top and write the number, while with the text I do have to worry about the proper formatting, typo, etc - and my god if I every will have to work with non-english descriptions, having trouble inputting character and sometimes even displaying them correctly, just to make the entry working.
In the editor, as I envision it, you would get instead a popup list of choices. But in order for the editor to know that it is not *just* a number, but rather a number out of a small possible range of choices and with semantic content, the information must be available somewhere -- one solution: stuff it in a metadata node apart from the data.
FaalagornI also like Moa's idea of making the XMLs to work under any browser and I don't see an issue with descriptions placed as a field for the first entry (though I have to admit, I prefer them being a comment as far as readability goes. But I understand that using comments as description may bring some troubles later on)
I also like Moa's idea; very much. There are a couple of things more that the tools, as I envision them, would do (e.g. handle upgrades more gracefully), but all in all, shifting the burden from code to data, is imho a clear win. But there is still a burden to be met nonetheless.
As far as the specific idea, it does not matter much where the descriptions are put, whether in attributes or in tags -- although I would argue that the latter is better, the reasons are not decisive. The main point however. is that they should be either in attributes or in tags; they can also be in comments, especially as it enhances human readability, but if they are to be consumed by the application (say to provide help in... an xml editor) then they must be either in attributes or tags.
Report message to a moderator
|
Private 1st Class
|
|
|
|
Goto Forum:
Current Time: Thu Apr 18 19:32:08 GMT+3 2024
Total time taken to generate the page: 0.01626 seconds
|
Blog
Search
Help
Members
Register
Login
Home
. I was looking at the source and gamedir SVNs and didn't knew there was a separate one for the XML editor. There is some confusion about the SVNs unfortunately 