12 replies to this topic

[mmeier]

Hey all,

 

as I've already stated in my New Arrivals thread, I'm currently building an automated checker for Sins mods. I would like to use this thread to gather a few ideas from the resident modders and from time to time ask questions on how Sins mods work.

 

At the moment, the app is written in Python, but that might change soon, as I've recently learned that that particular language is incapable of multithreading(aka using more than one cpu core). I would be interested to know how comfortable you all would be with a command line program without a GUI.

 

The following features are currently implemented:

• Syntax checking for nearly all Sins files in text form
• Checking of *.manifest and *.str files for duplicate entries

Future plans include:

• Check if referenced strings are present in the .str file
• Check if referenced entities exist and have the correct type (e.g. is the entity referenced in an Ability present and is it a Buff?)
• Check if referenced meshes/textures/audio files exist
• Check if all referenced files have an entry in their respective manifest file
• Check if a referenced brush has an entry in a brush file
• Check if a given file is actually referenced anywhere or can be removed

I would be happy to accept additional feature requests, if you can think of additional checks or just general drudgework tasks where you might think "A monkey could do this!".

 

Greetings,

Michael

[베이클라이트]

Kuhle Sache.

[Lavo]

For buff files, check that their formatting is right, and also do a check for whether the number of X actions (ex. numInstantActions) is true or not (ex. there is a count of 2, but there is only one instantAction).

[sloosecannon]

Hey all,

 

as I've already stated in my New Arrivals thread, I'm currently building an automated checker for Sins mods. I would like to use this thread to gather a few ideas from the resident modders and from time to time ask questions on how Sins mods work.

 

At the moment, the app is written in Python, but that might change soon, as I've recently learned that that particular language is incapable of multithreading(aka using more than one cpu core). I would be interested to know how comfortable you all would be with a command line program without a GUI.

 

The following features are currently implemented:

• Syntax checking for nearly all Sins files in text form
• Checking of *.manifest and *.str files for duplicate entries

Future plans include:

• Check if referenced strings are present in the .str file
• Check if referenced entities exist and have the correct type (e.g. is the entity referenced in an Ability present and is it a Buff?)
• Check if referenced meshes/textures/audio files exist
• Check if all referenced files have an entry in their respective manifest file
• Check if a referenced brush has an entry in a brush file
• Check if a given file is actually referenced anywhere or can be removed

I would be happy to accept additional feature requests, if you can think of additional checks or just general drudgework tasks where you might think "A monkey could do this!".

 

Greetings,

Michael

I would be happy to help and contribute my Java code if you're wanting to switch languages. I'm still working on it (Making a seperate class for every BuffInstantActionType may not have been the best move, in hindsight...) but it will provide the ability to do pretty much any of that.

[mmeier]

For buff files, check that their formatting is right, and also do a check for whether the number of X actions (ex. numInstantActions) is true or not (ex. there is a count of 2, but there is only one instantAction).

 

Yepp, it can already do all of those things, including the list-length-check. Only problem at the moment is that it's mostly disconnected pieces of code.

 

I would be happy to help and contribute my Java code if you're wanting to switch languages. I'm still working on it (Making a seperate class for every BuffInstantActionType may not have been the best move, in hindsight...) but it will provide the ability to do pretty much any of that.

Thanks a lot! I would really like to have a look at your code. I would especially be interested in which Java library you used to parse the Sins text files. I used pyparsing, which is great for something like this, but when I looked around yesterday after finding out that Python doesn't do multithreading, I did only find Java parsing libraries geared towards building language parsers.

 

And yeah, I know about the BuffInstantActions. Luckily, I didn't have to do one class per type, mostly because Python has multiple inheritance. Plus, multiple Actions have different names but the same content, thus they can be combined in a single class. But I still have a lot of classes. I think I've already added you to the bitbucket repo, have a look at the classes.uxf file (with e.g. UMLet) in the design directory, that's my current architecture. Definately not a thing of beauty. ;-)

 

Greetings,

Michael

[sloosecannon]

Yepp, it can already do all of those things, including the list-length-check. Only problem at the moment is that it's mostly disconnected pieces of code.

 

Thanks a lot! I would really like to have a look at your code. I would especially be interested in which Java library you used to parse the Sins text files. I used pyparsing, which is great for something like this, but when I looked around yesterday after finding out that Python doesn't do multithreading, I did only find Java parsing libraries geared towards building language parsers.

 

And yeah, I know about the BuffInstantActions. Luckily, I didn't have to do one class per type, mostly because Python has multiple inheritance. Plus, multiple Actions have different names but the same content, thus they can be combined in a single class. But I still have a lot of classes. I think I've already added you to the bitbucket repo, have a look at the classes.uxf file (with e.g. UMLet) in the design directory, that's my current architecture. Definately not a thing of beauty. ;-)

 

Greetings,

Michael

You may be disappointed... I'm not using a library.

 

Since Sins code follows a very standardized pattern, I'm basically just reading files line-by-line with a stream. If I encounter unexpected input, my parser breaks, but it also says why and where. Otherwise it just continues on happily. That is kinda the cause for my problems with the BuffInstantActions though... I have to write the parser for each option by hand :/

 

 

Fortunately though, it should work fairly well once I finish all that. Just... having to do it in the first place is kinda excruciating.

[mmeier]

You may be disappointed... I'm not using a library.

 

Since Sins code follows a very standardized pattern, I'm basically just reading files line-by-line with a stream. If I encounter unexpected input, my parser breaks, but it also says why and where. Otherwise it just continues on happily. That is kinda the cause for my problems with the BuffInstantActions though... I have to write the parser for each option by hand :/

 

 

Fortunately though, it should work fairly well once I finish all that. Just... having to do it in the first place is kinda excruciating.

 

Yeah, I thought to do the same in the beginning. Fortunately, I remembered Context Free Gammars from my first semester of University and subsequently found found and used pyparsing. And yes, I think excruciating fits perfectly well, especially considering that the Ironclad guys don't seem too fond of consistency...in some entities, some identifier are capitalized, in others they are not, in some the description string is called "desc" in others "description"...drove me crazy sometimes. And those instant actions were actually the first time I stopped and considered if I should can the idea.

 

Here is an example of how my parsers with pyparsing look like, with the brushes file as an example:

g_pixelBox = pp.Group(
        co.g_linestart
        + pp.Keyword("pixelBox")("identifier")
        + co.g_space
        + co.g_pos4("box")
        + co.g_eol
        )("pixelBox")
g_brushDef = (
        co.genStringAttrib("fileName")("texture")
        + g_pixelBox
        )
g_margin = (
        co.genIntAttrib("left")("left")
        + co.genIntAttrib("right")("right")
        + co.genIntAttrib("top")("top")
        + co.genIntAttrib("bottom")("bottom")
        )
g_margins = (
        pp.Group(
            co.genHeading("stretchMargins")
            + g_margin
            )("stretch")
        + pp.Group(
            co.genHeading("contentMargins")
            + g_margin
            )("content")
        )
g_states = (
        pp.Group(
            co.genHeading("Disabled")
            + g_brushDef
            )("disabled")
        + pp.Group(
            co.genHeading("Pressed")
            + g_brushDef
            )("pressed")
        + pp.Group(
            co.genHeading("CursorOver")
            + g_brushDef
            )("cursorOver")
        + pp.Group(
            co.genHeading("Focused")
            + g_brushDef
            )("focused")
        + pp.Group(
            co.genHeading("Normal")
            + g_brushDef
            )("normal")
        )
g_brushSimple = (
        co.genKeyValPair("content","Simple")("brushType")
        + pp.Group(g_brushDef)("def")
        )
g_brushMargins = (
        co.genKeyValPair("content","Margins")("brushType")
        + g_margins
        + pp.Group(g_brushDef)("def")
        )
g_brushStates = (
        co.genKeyValPair("content","States")("brushType")
        + g_states
        )
g_brushMarginsStates = (
        co.genKeyValPair("content","MarginsAndStates")("brushType")
        + g_margins
        + g_states
        )
g_brush = pp.Group(
        co.genHeading("brush")
        + co.genStringAttrib("name")("name")
        + pp.Or([g_brushSimple,g_brushMargins,g_brushStates,
            g_brushMarginsStates])
        )
g_BrushFile = (
        co.g_txt
        + co.genBoolAttrib("onlyLoadOnDemand")("loadOnDemand")
        + co.genListAttrib("brushCount",g_brush)("brushes")
        + pp.StringEnd().suppress()
        )

Edit:

 

This is everything that's needed to parse a *.brushes file. Given such a file, I now only need to call

g_BrushFile.parseFile("testfile")

and get the parsing result as a disctionary/hashmap.

[mmeier]

Okay, I've just pushed a number of commits to Bitbucket, sloose. Its the first useable version. If you would like to try it, check out the repo. The things currently working are:

 

• General checks of manifests with checking if all the entries actually exist as files and searching for duplicate entries
• General checks of string files, with checking for duplicates

 

To use the current version, you will need the following dependencies:

• At least Python 3.4
• https://pypi.python....ortedcontainers
• https://pypi.python....gcomplete/1.0.0
• https://pypi.python....pyparsing/2.0.3

With all of those, you can then call the src/soase.py file. Enter it with --help to see the usage message. Lets assume you want to check all SotP manifest and string files, you would call pysoase like this:

soase.py check general path/to/sotp/*.manifest path/to/sotp/String/*.str

Then, it should output a few warnings about those files with the current dev repository of sotp.

 

EDIT:

 

Ehm, if you are using Windows, the *.manifest/*.str probably won't work...I don't too much about the Windows console. You might have to enumerate each file separately. You can name any str/manifest file on the command line, but it won't work for other filetypes or directories right now.

[sloosecannon]

Okay, I've just pushed a number of commits to Bitbucket, sloose. Its the first useable version. If you would like to try it, check out the repo. The things currently working are:

 

• General checks of manifests with checking if all the entries actually exist as files and searching for duplicate entries
• General checks of string files, with checking for duplicates

 

To use the current version, you will need the following dependencies:

• At least Python 3.4
• https://pypi.python....ortedcontainers
• https://pypi.python....gcomplete/1.0.0
• https://pypi.python....pyparsing/2.0.3

With all of those, you can then call the src/soase.py file. Enter it with --help to see the usage message. Lets assume you want to check all SotP manifest and string files, you would call pysoase like this:

soase.py check general path/to/sotp/*.manifest path/to/sotp/String/*.str

Then, it should output a few warnings about those files with the current dev repository of sotp.

 

EDIT:

 

Ehm, if you are using Windows, the *.manifest/*.str probably won't work...I don't too much about the Windows console. You might have to enumerate each file separately. You can name any str/manifest file on the command line, but it won't work for other filetypes or directories right now.

I shall try that

 

I use Windows but........ I also use Cygwin for my shell

So the globbing should work fine

 

I think I'll look into those grammar parsers, although it appears I'm basically doing just that, manually... (probably a Bad Idea)

[Fleet Admiral agigabyte]

What? Oh, yeah, I understand this.

[mmeier]

I shall try that

 

I use Windows but........ I also use Cygwin for my shell

So the globbing should work fine

 

I think I'll look into those grammar parsers, although it appears I'm basically doing just that, manually... (probably a Bad Idea)

 

Ahhhh cygwin. The day I learned of it's existence was the day I could finally work comfortably in Windows. :-)

 

Context free grammars and their associated parsers are really nice. At first, I thought to do all the parsing manually too - but then I thought "Hey, I can't be the first guy who needs to parse a well structured text file" and I wasn't. One still needs to supply the grammar definition, but at least I didn't need to write the whole mechanics of actually doing the parsing. When you add a few functions to make things like "one line with two strings separated by a space, the second one in quotes", which show up often, simple to use, you can write the parsers for all the Sins files in no time. Took me only about 2 weeks for all the entity types. I've only recently slowed down when I started to write the actual logic of checking...mostly because I decided to write Unit tests. If you want to have a laugh, compare the line counts of the actual source code and the tests living under src/pysoase/tests. :-D

 

What? Oh, yeah, I understand this.

But that's exactly the reason I'm writing this - so you don't need to understand it. Or, well, you won't need to understand it if I ever get around to writing a nice, usable user interface. ;-)

[mmeier]

Hey people,

 

a question for the modders among you: Lets say you use an entity from the vanilla game in your mod. Would that vanilla entity need to be in your mod's  entity.manifest?

 

Greetings,

Michael

[sloosecannon]

Hey people,

a question for the modders among you: Lets say you use an entity from the vanilla game in your mod. Would that vanilla entity need to be in your mod's entity.manifest?

Greetings,
Michael

Yes it would, but it would not need to be in the mod directory

Sent from my Nexus 6 using Tapatalk