Dumping skills without skill dump setting :)

[Muzzy 43]

So, I wanted to have a list of my character's skills and the normal way to do this would be using the skill dump. However, that would require logging in and I'm not feeling like doing that, so I had to do it another way. Posting my method here in case anyone finds it interesting.
 

cat ~/wurm/players/muzzy/logs/_Skills.* | perl -e 'while(<>) { /[^]]*\] (.*) increased .* to (.*)/ and do { if ($2>$skills{$1}) { $skills{$1} = $2; } }}; @ss = reverse sort {$skills{$a} <=> $skills{$b}} keys %skills; foreach my $skill (@ss) { print "$skills{$skill} $skill\n"; }'

This will print out in a nice list of skills, sorted by highest skill first. It will list the highest values the skills have ever been, so in case of lost skills it will not necessarily be the current value.
 
The output will look like this (only pasting the first few lines because the full dump is kinda long)

72.0259 Carpentry62.0083 Digging56.9122 Blacksmithing56.8352 Miscellaneous items56.548 Fighting56.4393 Pickaxe

Since there are timestamps in those logs it would be easy to make some graphs of skill progress too based on the logs, I might get into that later today if I get too bored in the evening. The existing skill graphing solutions seem to require the skill dumps, at least I couldn't find anything that would make graphs from the raw log files.

[lpleroylp 59]

Nice one but shouldnt you post this in suggestions thread instead other then that.

 

+1

Edited April 14, 2015 by lpleroylp

[Drastox 332]

Its not entirely a suggestion really. More or less showing some brilliant skillz in coding

[Muzzy 43]

Okay, so I got bored quite fast and decided to tinker with the graph generation. After checking out various plotting solutions, I realized I already had a python library for this purpose installed so I used python and matplotlib for the job.

 

Trying to decide which skills to put in the graph made me realize how pointless all the skills really are. Which ones of them are important? Trying to fit all the skills in there ended up with a horrible mess so I picked a few random ones.

 

Here's the full source code to the python script I used. With hardcoded paths because I'm not writing this as a tool for other people, I just wanted some graphs to reflect on the past few months of my life.

import matplotlib.pyplot as plt

from matplotlib.dates import date2num

from glob import glob

import datetime

import re


def process_skill_log(file):

    data = {};

    previoustime = None

    for line in [x.strip() for x in file]:

        if line.startswith("Logging started"):

            # Assume YYYY-mm-dd at exact offsets

            day = datetime.date(int(line[16:20]), int(line[21:23]), int(line[24:26]))

        m = re.match('[^]]*\] (.*) increased .* to (.*)', line)

        if m:

            # Assume [hh:mm:ss] at beginning of line

            time = datetime.datetime.combine(day, datetime.time(int(line[1:3]), int(line[4:6]), int(line[7:9])))

            # Handle timestamp wrapping

            if (previoustime != None and time < previoustime):

                time += datetime.timedelta(days=1)

            previoustime = time


            skillname = m.group(1)

            skillvalue = float(m.group(2))


            if not skillname in data:

                data[skillname] = {'dates': [], 'values': []};

            data[skillname]['dates'].append(date2num(time))

            data[skillname]['values'].append(skillvalue)


    return data


skilldata = {}


for x in glob('/Users/muzzy/wurm/players/muzzy/logs/_Skills.*'):

    newdata = process_skill_log(open(x))

    for skill in newdata:

        if not skill in skilldata:

            skilldata[skill] = newdata[skill]

        else:

            skilldata[skill]['dates'].extend(newdata[skill]['dates'])

            skilldata[skill]['values'].extend(newdata[skill]['values'])


for skill in ['Carpentry', 'Fighting', 'Blacksmithing', 'Cloth tailoring', 'Leatherworking', 'Woodcutting', 'Weapon smithing']:

    plt.plot_date(skilldata[skill]['dates'], skilldata[skill]['values'], '-', label=skill, xdate=True, linewidth=2)


plt.legend(loc="upper left")

plt.grid(True)


plt.show()

[Cally 96]

I hope someone does take this and code it into something like the wurm assistant thing -- it's a neat little tool! 

[Warlander 2,730]

I hope someone does take this and code it into something like the wurm assistant thing -- it's a neat little tool!

Even better - Wurm Assistant mixed with Wurm Online Skillcompare.

[Stimrol 77]

Very nice commands, will be bookmarking this and using the cat command from time to time. But is it possible to let the code work with skills that have comma seperator instead of periods. All my skills are saved and showed with 10,5 not 10.5. Would be very nice if this is some easy fix.

[Cenotaph 118]

That graph looked awesome, if I only knew what the hell that was and how to use it

I guess google when I get back from work will do the trick as usual.

[Muzzy 43]

Well yeah, it's going to be difficult to use for non-programmers since I'm not really releasing user friendly tools, just dumping what I made so that other programmers can take it and run with it. I should've also added a license there or something, let's just say it's public domain for now, use as you wish and no attribution required.

 

@Stimrol, the skills in your skill log have different format? Well, uh, wow. If that's really the case, then this line:

skillvalue = float(m.group(2))

needs to be changed like this:

skillvalue = float(m.group(2).replace(",","."))

I obviously only tested my script with my own log files, and I have no idea how it's going to work with other people. There might be some other surprises as well for some people, i.e. if the number formats change based on locale then the datestamps might be messed too, who knows. I'm just dropping the code here because it worked for me. I'm not really planning to write a tool that others could use or maintain the code, sorry.

[asdf 297]

Here's an alternative shellscript, uses unix programs rather than perl so it should work in different locales (like on Stimrol's system):

cat wurm/players/Asdf/logs/_Skills.20* |  perl -ne '/[^]]*\] (.*) increased .* to (.*)/ and print "$2 $1\n";' | sort -k 2,3 -k 1nr | uniq -f 1 |sort -rn

Still uses perl for the regexp because sed's regexps are way slow

 

As for the python code, the way to make it work in other locales is to change the

skillvalue = float(m.group(2))

line to say this instead:

skillvalue = locale.atof(m.group(2))

and also add this at the top of the file:

import locale

(this will work on any locale, with both . and , and also others if they're used on some systems).

 

Also the python script will break if you have more than a year worth of logs, because the input files are never sorted, so it'll first add -01.txt to the graph, then -10.txt, etc, borking the graph. Need to sort the files first.

Change this line:

for x in glob('/Users/muzzy/wurm/players/muzzy/logs/_Skills.*'):

to say:

def sortfilename(filename):
    year, month = re.search("_Skills.(\d+)-(\d+)", filename).groups()
    return int(year), int(month)

for x in sorted(glob('wurm/players/Asdf/logs/_Skills.*'), key=sortfilename):

Complete code with both changes:

import matplotlib.pyplot as plt
from matplotlib.dates import date2num
from glob import glob
import datetime
import locale
import re

def process_skill_log(file):
    data = {};
    previoustime = None
    for line in [x.strip() for x in file]:
        if line.startswith("Logging started"):
            # Assume YYYY-mm-dd at exact offsets
            day = datetime.date(int(line[16:20]), int(line[21:23]), int(line[24:26]))
        m = re.match('[^]]*\] (.*) increased .* to (.*)', line)
        if m:
            # Assume [hh:mm:ss] at beginning of line
            time = datetime.datetime.combine(day, datetime.time(int(line[1:3]), int(line[4:6]), int(line[7:9])))
            # Handle timestamp wrapping
            if (previoustime != None and time < previoustime):
                time += datetime.timedelta(days=1)
            previoustime = time

            skillname = m.group(1)
            skillvalue = locale.atof(m.group(2))

            if not skillname in data:
                data[skillname] = {'dates': [], 'values': []};
            data[skillname]['dates'].append(date2num(time))
            data[skillname]['values'].append(skillvalue)

    return data

skilldata = {}

def sortfilename(filename):
    year, month = re.search("_Skills.(\d+)-(\d+)", filename).groups()
    return int(year), int(month)

for x in sorted(glob('wurm/players/Asdf/logs/_Skills.*'), key=sortfilename):
    newdata = process_skill_log(open(x))
    for skill in newdata:
        if not skill in skilldata:
            skilldata[skill] = newdata[skill]
        else:
            skilldata[skill]['dates'].extend(newdata[skill]['dates'])
            skilldata[skill]['values'].extend(newdata[skill]['values'])

for skill in ['Carpentry', 'Fighting', 'Blacksmithing', 'Cloth tailoring', 'Leatherworking', 'Woodcutting', 'Weapon smithing']:
    plt.plot_date(skilldata[skill]['dates'], skilldata[skill]['values'], '-', label=skill, xdate=True, linewidth=2)

plt.legend(loc="upper left")
plt.grid(True)

plt.show()

Edited April 14, 2015 by asdf

[Muzzy 43]

@asdf, Nice! I wondered about the filename sorting, but since it didn't pose a problem for me I didn't think into it too deeply.

 

Also I completely forgot about uniq -f, I totally would have done it that way myself too if only I had realized to man uniq. Your version is definitely better than my ugly perl hack

[Stimrol 77]

Thanks both for the help, yes the files are written with dots in my local settings, becaue we use . as 1000 seperator and , as extra letters.

 

I am so thankfull for this Muzzy because I have not been able to use this wurmassistant tool because of running linux based system.

 

 

This explains why my result with the replace of , to . where all messy in the output.

Also the python script will break if you have more than a year worth of logs, because the input files are never sorted, so it'll first add -01.txt to the graph, then -10.txt, etc, borking the graph. Need to sort the files first.

Change this line:

for x in glob('/Users/muzzy/wurm/players/muzzy/logs/_Skills.*'):

[Stimrol 77]

I tried again and still got a error:

Traceback (most recent call last):

  File "skill_graph.py", line 41, in <module>

    newdata = process_skill_log(open(x))

  File "skill_graph.py", line 25, in process_skill_log

    skillvalue = locale.atof(m.group(2))

  File "/usr/lib/python2.7/locale.py", line 316, in atof

    return func(string)

ValueError: invalid literal for float(): 2,000

I then changed the skillvalue line to this:

skillvalue = float(m.group(2).replace(',', '.'))

and then this works.

 

 

Thanks again

Edited April 14, 2015 by Stimrol

[asdf 297]

Eh, I dunno then, probably some interesting combination of locales where wurm thinks it's in one that uses a comma, and the script thinks it should use the dot. String methods are fine

[Muzzy 43]

Locales are always massive pain

[Stimrol 77]

Next is of course to add check marks to all the available skills so there would be no need to save the file all the time.

 

Feel little bad always asking for more hehe.

[Muzzy 43]

@Stimrol, Instead of specifying an array of skills, just use this instead and cry in horror:

for skill in skilldata.keys():

So, check marks, eh? I suppose I'll check the documentation to see if it's easy to make an UI for that, but unless it's super easy to do I'm not going to bother with it.

[Muzzy 43]

It looks like UI would be too much trouble. I mean, I managed to make the checkmarks sort of fine, but there are just too many skills to fit in the screen at once and making a scrollable list of checkmarks would require more than the matplotlib widgets allows for. Embedding the whole thing inside a gtk application and making a scrollable list for skills would work, but at the same time it would also require way too much effort because it would mean losing the built-in UI with the save button etc (at least I think, didn't try embedding).

 

The second best alternative to editing the source code would be taking the list of skills from command line arguments. But I'm not going to bother making the commandline parsing either because it doesn't significantly improve the tool by itself, I'd just end up being consumed by feature creep since things like player name should be made a commandline argument too and maybe date range and perhaps the skill selection should happen automatically based on threshold (e.g. display all skills that have ended up above 50), maybe an option for logarithmic scale, etc etc etc...

 

I'll let someone else tinker with it, I'm done with the featureset unless I can think of something new that I want to see about my own data