OGame Automizer Development Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

Welcome on the OGame Automizer Development/Support Forum

If this is your first visit, be sure to check out the
http://ogameautomizer.com/doc/manual.html : A guide written by OA users with consultation of OA author (En)

http://ogameautomizer.com/doc/manual_de.html : A guide written by OA users with consultation of OA author (Benutzerschnittstellenhandbuch) (GE)

http://ogameautomizer.com/doc/scripting_manual.html : OA Scripting Manual (En)

Pages: [1] 2 3 4

Author Topic: Import OGame XML files once a week  (Read 13581 times)

0 Members and 1 Guest are viewing this topic.

Devil's Hand

  • Administrator
  • Hero Member
  • *****
  • Offline Offline
  • Posts: 2,523
    • View Profile
Import OGame XML files once a week
« on: July 04, 2012, 10:08:32 PM »

Since OGame 4.1.1, the data about players/planets-moons in universe/scores/ranks is available publicly: http://board.origin.ogame.de/board6-origin/board38-tools-scripts-skins/3927-ogame-api/

Below the script which once a week (data about planets in universe is update once a week), just after XML files been updated by OGame, imports data to OA internal Database.

Create new OA Script (right click on the scripts Tab)
Rename it to something like 'Import OGame XMLs',
Enable the option 'Execute after OA initialization',
Enable the option 'Restart the script on error',
In the code editor (delete template code: select all text (CTRL+A) than Backspace), paste the following code:
Code: [Select]
// This script imports to OGameAutomizer Database data from OGame XML files available since OGame v4.1 in all universes
// To get best of it, it should be executed once a week, just after XMLs update, it's done automaticlly by enabling 'automatedImport' option
// Caution: this script is very-very Database intensive during import execution, it may also increase overall OA memory usage !
// Known error: "Script XXX has been stopped, Throwable: Premature EOF" is caused by your network issue (connection lost, timeout etc.)
// to bypass the error above, enable 'Restart the script on error'

// -------- Config starts --------
def automatedImport = YES   // YES = checks every hour if it can get the latest version, NO = provides you a GUI to start import manually
def actionsAfterImport = { ->   // actions to execute after full and successful import(once a week), e.g. start Farming Bot, launch another script, send mail/SMS etc.
  //startFarmingBot(FastFarming=true)
  //startScript("Farming Bot")
  //startScript("Hunter")
  //...
  printOut "All actions done."
}
def verbose = YES   // YES = Print additional outputs, NO = don't print (less CPU usage)
// -------- Config ends -----------

def playersXMLRecords, universeXMLRecords, alliancesXMLRecords, pHighscoresXMLRecords, aHighscoresXMLRecords = null
def init = { ->
  playersXMLRecords = new XmlParser().parse("http://"+getOGameAddress()+"/api/players.xml").player
  universeXMLRecords = new XmlParser().parse("http://"+getOGameAddress()+"/api/universe.xml").planet
  alliancesXMLRecords = new XmlParser().parse("http://"+getOGameAddress()+"/api/alliances.xml").alliance
  pHighscoresXMLRecords = new XmlParser().parse("http://"+getOGameAddress()+"/api/highscore.xml?category=1&type=0").player
  aHighscoresXMLRecords = new XmlParser().parse("http://"+getOGameAddress()+"/api/highscore.xml?category=2&type=0").alliance
}

def importPlayerData = { id ->
  def p = playersXMLRecords.findAll{ it.@id == id }
  if(p.size() > 0) {
    def u = universeXMLRecords.findAll{ it.@player == p[0].@id }
    for(int i = 0; i < u.size(); i++){
      def a = alliancesXMLRecords.findAll{ it.@id == p[0].@alliance }
      if(pHighscoresXMLRecords.findAll{ it.@id == p[0].@id }[0] != null){
        if(verbose) printOut p[0].@name + "\t" + (p[0].@status == null ? "active" : p[0].@status) + "\t" +
          (a[0] == null ? "no ally" : a[0].@tag + "/" + a[0].@name) + "\t" +
          "planet_name="+u[i].@name + "\t[" + u[i].@coords + "]\t" +
          (pHighscoresXMLRecords.findAll{ it.@id == p[0].@id }[0] == null ? "" : "p_rank="+pHighscoresXMLRecords.findAll{ it.@id == p[0].@id }[0].@position + "\t" + "p_score="+pHighscoresXMLRecords.findAll{ it.@id == p[0].@id }[0].@score) + "\t" +
          (a[0] == null || aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0] == null ? "no ally" : "a_rank="+aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0].@position + "\t" + "a_score="+aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0].@score)
       
        addPlayer2DB (p[0].@name, Integer.parseInt(p[0].@id), Short.parseShort(pHighscoresXMLRecords.findAll{ it.@id == p[0].@id }[0].@position),
          (a[0] == null ? "" : a[0].@name),
          (a[0] == null || aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0] == null ? (short)0 : Short.parseShort(aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0].@position)),
          (a[0] == null ? (int)0 : Integer.parseInt(a[0].@id)),
          (a[0] == null ? (short)0 : (short)alliancesXMLRecords.findAll{ it.@id == a[0].@id }[0].player.size()),
          u[i].@name, Boolean.FALSE, Coordinates.getCoordinates(u[i].@coords).galaxy, Coordinates.getCoordinates(u[i].@coords).system, Coordinates.getCoordinates(u[i].@coords).planet,
          (p[0].@status == null ? "" : p[0].@status)
        )
      } else {
        if(verbose) printOut "Ignoring: "+p[0].@name+" : "+u[i].@coords+" (ok if Admin or Banned)"
        break
      }
     
      // Print&Import Moon, if any
      if(universeXMLRecords.findAll{ it.@player == p[0].@id }[i].moon.@name.size() > 0){
        if(verbose) printOut p[0].@name + "\t" + (p[0].@status == null ? "active" : p[0].@status) + "\t" +
          (a[0] == null ? "no ally" : a[0].@tag + "/" + a[0].@name) + "\t" +
          "planet_name="+u[i].moon.@name[0] + "\t[" + u[i].@coords+"] Moon\t" +
          (pHighscoresXMLRecords.findAll{ it.@id == p[0].@id }[0] == null ? "" : "p_rank="+pHighscoresXMLRecords.findAll{ it.@id == p[0].@id }[0].@position + "\t" + "p_score="+pHighscoresXMLRecords.findAll{ it.@id == p[0].@id }[0].@score) + "\t" +
          (a[0] == null || aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0] == null ? "no ally" : "a_rank="+aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0].@position + "\t" + "a_score="+aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0].@score)
       
        addPlayer2DB (p[0].@name, Integer.parseInt(p[0].@id), Short.parseShort(pHighscoresXMLRecords.findAll{ it.@id == p[0].@id }[0].@position),
          (a[0] == null ? "" : a[0].@name),
          (a[0] == null || aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0] == null ? (short)0 : Short.parseShort(aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0].@position)),
          (a[0] == null ? (int)0 : Integer.parseInt(a[0].@id)),
          (a[0] == null ? (short)0 : (short)alliancesXMLRecords.findAll{ it.@id == a[0].@id }[0].player.size()),
          u[i].moon.@name[0], Boolean.TRUE, Coordinates.getCoordinates(u[i].@coords).galaxy, Coordinates.getCoordinates(u[i].@coords).system, Coordinates.getCoordinates(u[i].@coords).planet,
          (p[0].@status == null ? "" : p[0].@status)
        )
      }
    }
  }
}

def fileName = getOGameAddress()+"_Import_XMLs"
if(automatedImport){
  def lastImport = timestamp
  if(new File(fileName).exists() && readFile(fileName).size() != 0) lastImport = Long.parseLong(readFile(fileName).trim())
  else writeFile(fileName, ""+lastImport)
  while(true){
    def xmlUpdateTimestamp = 1000*Long.parseLong(new XmlParser().parse("http://"+getOGameAddress()+"/api/universe.xml").planet[0].parent().@timestamp)
    if(lastImport < xmlUpdateTimestamp){
      printOut "Start importing..."
      lastImport = timestamp
      init()
      writeFile(fileName, ""+lastImport)
      for(i in 0..playersXMLRecords.size()-1){
        if(verbose) printOut "adding #"+(i+1)+", player_id="+playersXMLRecords[i].@id
        importPlayerData playersXMLRecords[i].@id
      }
      printOut "Imported "+playersXMLRecords.size()+" players from OGame XMLs"
      printOut "Imported "+universeXMLRecords.size()+" planets(moons not counted) from OGame XMLs"
      actionsAfterImport()
    }
    printOut "Next XMLs update will be in: "+Time.getTime(xmlUpdateTimestamp+7*24*3600*1000-timestamp)+", sleeping..."
    sleep xmlUpdateTimestamp+7*24*3600*1000-timestamp+new Random().nextInt(3600000)
  }
} else {
  %ScriptMode%
  playersXMLRecords = new XmlParser().parse("http://"+getOGameAddress()+"/api/players.xml").player
  def running = false
  def b = null
  new SwingBuilder().edt {
    def frame = frame(title: "Importing OGame Data... ("+getOGameAddress()+" "+getLogin()+")",
          defaultCloseOperation:JFrame.DO_NOTHING_ON_CLOSE,
          windowClosing : { event -> running = false; dispose() },
          location: [(int)(java.awt.Toolkit.getDefaultToolkit().getScreenSize().width-500),
                     (int)(java.awt.Toolkit.getDefaultToolkit().getScreenSize().height-140)],
          size: [500, 100], show: true, alwaysOnTop: true)
    {
      gridLayout(cols: 1, rows: 2)
      b = button( "Start",
        actionPerformed: {
          running = !running
          if(running) Thread.start{
            printOut "Start importing..."
            b.text = "Stop"
            progress.value = 0
            init()
            writeFile(fileName, timestamp)
            for(i in 0..playersXMLRecords.size()-1){
              if(!running) return
              if(verbose) printOut "adding #"+(i+1)+", player_id="+playersXMLRecords[i].@id
              importPlayerData playersXMLRecords[i].@id
              progress.value = i
            }
            printOut "Imported "+playersXMLRecords.size()+" players from OGame XMLs"
            printOut "Imported "+universeXMLRecords.size()+" planets(moons not counted) from OGame XMLs"
            running = false
            b.text = "Start"
          }
          else b.text = "Start"
        }
      )
      progress = progressBar(minimum: 0, maximum: playersXMLRecords.size()-1)
    }
  }
}

Caution: this script is very-very Database (CPU/disk IO) intensive during import execution, it may also increase overall OA memory usage !

Regards,
« Last Edit: August 07, 2015, 07:38:01 PM by Devil's Hand »
Logged

fsueer

  • Newbie
  • *
  • Offline Offline
  • Posts: 31
    • View Profile
Re: Import OGame XML files once a week
« Reply #1 on: March 11, 2013, 12:22:49 AM »

Hu, i started to use this script since today but always get this error.

And i hope i use this script for the right purpose.

Will this update the playerdatabase netxt to hunter in OA? Hope so^^ Because the to scann the whol universe via Player Database within OA takes too long.

Now to the Error

00:17:43 : java.io.IOException: Premature EOF
00:17:43 :    at groovy.util.XmlParser.parse(XmlParser.java:192)
00:17:43 :    at groovy.Update Universe$_run_closure95_closure99.doCall(Update Universe:896)
00:17:43 :    at groovy.Update Universe$_run_closure95.doCall(Update Universe:812)
00:17:43 :    at groovyx.gpars.GParsPool$_withExistingPool_closure1.doCall(GParsPool.groovy:170)
00:17:43 :    at groovyx.gpars.GParsPool.withExistingPool(GParsPool.groovy:169)
00:17:43 :    at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:141)
00:17:43 :    at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:117)
00:17:43 :    at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:96)
00:17:43 :    at groovy.Update Universe.run(Update Universe:811)
00:17:43 :
00:17:43 : Script uni118.ogame.de XXXX has been stopped, Throwable: Premature EOF

The script will be restarted...    5    4    3    2    1


and so it takes every day^^

Hope for help many thx
Logged

Devil's Hand

  • Administrator
  • Hero Member
  • *****
  • Offline Offline
  • Posts: 2,523
    • View Profile
Re: Import OGame XML files once a week
« Reply #2 on: March 13, 2013, 07:47:45 PM »

try to print:
Code: [Select]
printOut "http://"+getOGameAddress()+"/api/players.xml"than copy and paste it in your browser, do you receive an XML file ?
Logged

troniac

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 358
  • Rue de Bruxelles, 5000 Namur, Belgium
    • View Profile
Re: Import OGame XML files once a week
« Reply #3 on: May 07, 2013, 02:33:46 PM »

I copied the SCript into an Empty new Script Tab.
Then I executed it and the following will print to the Console. And it stopped working.

Code: [Select]
14:27:01 : Start importing...
14:27:13 : adding #1, player_id=1
14:27:13 : Ignoring: Legor : 1:1:2 (ok if Admin or Banned)
14:27:13 : adding #2, player_id=77
14:27:14 : Optimus Prime vi Infineon/Infineon planet_name=Fortress Maximus [1:39:3] p_rank=1163 p_score=2162102 a_rank=236 a_score=4538104

Code: [Select]
        addPlayer2DB (p[0].@name, Integer.parseInt(p[0].@id), Short.parseShort(pHighscoresXMLRecords.findAll{ it.@id == p[0].@id }[0].@position),
          (a[0] == null ? "" : a[0].@name),
          (a[0] == null || aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0] == null ? (short)0 : Short.parseShort(aHighscoresXMLRecords.findAll{ it.@id == a[0].@id }[0].@position)),
-->          (a[0] == null ? (short)0 : Short.parseShort(a[0].@id)),
-->          (a[0] == null ? (short)0 : (short)alliancesXMLRecords.findAll{ it.@id == a[0].@id }[0].player.size()),
          u[i].@name, Boolean.FALSE, Coordinates.getCoordinates(u[i].@coords).galaxy, Coordinates.getCoordinates(u[i].@coords).system, Coordinates.getCoordinates(u[i].@coords).planet,
          (p[0].@status == null ? "" : p[0].@status)
        )

I found that this Lines in (short) Format gets out of range cause Alliance Score is 36.770 Points this is "out of Range" for short DataType. I can not use Ineger because the addPlayer2DB() Function doesn't accept Integer at this point! Please, can you modify this?

This java.lang.NumberFormatException makes this Script useless. :( May you fix the AddUser2DB Function and all other corresponding functions to have the possibility to give Integers instead of Shorts to the function? Or add another Function Signature. Thank you very much!
« Last Edit: August 27, 2013, 01:13:50 PM by troniac »
Logged
Je'sui Brussels

troniac

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 358
  • Rue de Bruxelles, 5000 Namur, Belgium
    • View Profile
Re: Import OGame XML files once a week
« Reply #4 on: August 27, 2013, 01:09:19 PM »

You can Reproduce this with the

Code: [Select]
// Short Test Script.
//--> Working
printOut Short.parseShort("32000")

//--> Exception
printOut Short.parseShort("36770")

//--> Solution
printOut Integer.parseInt("36770")

Output on Error Exception:
Code: [Select]
13:08:36 : Script Script_154742 has been stopped, Throwable: Value out of range. Value:"36770" Radix:10
13:08:36 : Below full stack trace:
13:08:36 :
13:08:36 : java.lang.NumberFormatException: Value out of range. Value:"36770" Radix:10
13:08:36 : at Script_154742.run(Script_154742:3)
« Last Edit: August 27, 2013, 01:12:04 PM by troniac »
Logged
Je'sui Brussels

Devil's Hand

  • Administrator
  • Hero Member
  • *****
  • Offline Offline
  • Posts: 2,523
    • View Profile
Re: Import OGame XML files once a week
« Reply #5 on: August 27, 2013, 11:17:23 PM »

Try this build: http://ogameautomizer.com/BetaVersion/OGameAutomizer.jar
+ Script above updated.

Regards,
« Last Edit: August 27, 2013, 11:23:10 PM by Devil's Hand »
Logged

troniac

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 358
  • Rue de Bruxelles, 5000 Namur, Belgium
    • View Profile
Re: Import OGame XML files once a week
« Reply #6 on: August 28, 2013, 09:19:35 AM »

Thank you very much!
It's working now!

For your Information. It would be faster if you execute the AddPlayer2DB Function only one time and first prepare a Data.Set in Memory and insert it into the Database at once and not for each line INSERT INTO
Logged
Je'sui Brussels

TheMann

  • Newbie
  • *
  • Offline Offline
  • Posts: 29
    • View Profile
Re: Import OGame XML files once a week
« Reply #7 on: December 03, 2013, 07:00:43 PM »

Hey guys can u help me i have run the script and i get is this when i put yes in config for automated update.. and nothing else.... does this mean its been updated?

Code: [Select]
17:53:44 : Next XMLs update will be in: 6d 20:13:50, sleeping.../code]
but when i do it manually i recieve this message..

Code: [Select]
19:02:57 : Start importing...
19:03:00 : The content of elements must consist of well-formed character data or markup.
19:03:00 : org.xml.sax.SAXParseException: The content of elements must consist of well-formed character data or markup.
19:03:00 : at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1231)
19:03:00 : at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
19:03:00 : at groovy.util.XmlParser.parse(XmlParser.java:198)
19:03:00 : at Import Uni DAte$_run_closure2.doCall(Import Uni DAte:22)
19:03:00 : at Import Uni DAte$_run_closure4_closure31_closure32_closure33.doCall(Import Uni DAte:119)
19:03:00 : at groovy.OAWrapper$CustomThread.run(OAWrapper.groovy:802)
« Last Edit: December 03, 2013, 08:08:22 PM by TheMann »
Logged

hannelore

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 50
    • View Profile
Re: Import OGame XML files once a week
« Reply #8 on: December 05, 2013, 11:41:10 AM »

Hi guys,


the database is not transferred to my hunter or player database. what am I doing wrong?


I get no error message ...

Quote
Next XMLs update will be in: 6d 02:03:06, sleeping...



...Problem has been solved by itself...
« Last Edit: December 11, 2013, 07:34:12 PM by hannelore »
Logged

NickyFam

  • Newbie
  • *
  • Offline Offline
  • Gender: Male
  • Posts: 1
    • View Profile
Import OGame XML files once a week
« Reply #9 on: December 24, 2015, 08:42:49 AM »

Is there any way to import from two data base like IMDB and Allocine at once? And mix together?
Logged

Phil

  • Newbie
  • *
  • Offline Offline
  • Posts: 4
    • View Profile
Re: Import OGame XML files once a week
« Reply #10 on: May 12, 2016, 05:47:01 PM »

anyome know why i am getting this error

16:43:33 : Script OGame API has been stopped, Throwable: DOCTYPE is disallowed when the feature "http://apache.org/xml/features/disallow-doctype-decl" set to true.
16:43:33 : Below full stack trace:
16:43:33 :
16:43:33 : org.xml.sax.SAXParseException; systemId: http://s122-en.ogame.gameforge.com/api/players.xml; lineNumber: 1; columnNumber: 10; DOCTYPE is disallowed when the feature "http://apache.org/xml/features/disallow-doctype-decl" set to true.
16:43:33 :    at groovy.util.XmlParser.parse(XmlParser.java:267)
16:43:33 :    at OGame_API_id_df684c_.run(OGame API:100)
Logged

wulf3010

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 447
    • View Profile
Re: Import OGame XML files once a week
« Reply #11 on: May 12, 2016, 07:39:36 PM »

got the same error. Have newest Java Runtime with win7
Logged

Phil

  • Newbie
  • *
  • Offline Offline
  • Posts: 4
    • View Profile
Re: Import OGame XML files once a week
« Reply #12 on: May 20, 2016, 07:02:06 PM »

Get anu advise on my above post?
Logged

AnonymousUser

  • Newbie
  • *
  • Offline Offline
  • Posts: 11
    • View Profile
Re: Import OGame XML files once a week
« Reply #13 on: July 18, 2016, 09:30:14 PM »

Same problem here, anyone ???

anyome know why i am getting this error

16:43:33 : Script OGame API has been stopped, Throwable: DOCTYPE is disallowed when the feature "http://apache.org/xml/features/disallow-doctype-decl" set to true.
16:43:33 : Below full stack trace:
16:43:33 :
16:43:33 : org.xml.sax.SAXParseException; systemId: http://s122-en.ogame.gameforge.com/api/players.xml; lineNumber: 1; columnNumber: 10; DOCTYPE is disallowed when the feature "http://apache.org/xml/features/disallow-doctype-decl" set to true.
16:43:33 :    at groovy.util.XmlParser.parse(XmlParser.java:267)
16:43:33 :    at OGame_API_id_df684c_.run(OGame API:100)
Logged

marsupilami

  • Newbie
  • *
  • Offline Offline
  • Posts: 27
    • View Profile
Re: Import OGame XML files once a week
« Reply #14 on: September 20, 2016, 10:43:58 PM »

Hi you need to change all http entry by https in the script

Running well and Database is filled by Ogame API
Logged
Pages: [1] 2 3 4
 

Page created in 0.064 seconds with 20 queries.