July 31, 2009

Reverse Engineering the EVE-Online Cache Files

So, I’m doing it after all. Reverse engineering the cache files. The format is actually surprisingly simple once you determine the identifiers of all the data fields (and their lengths). No reverse compiling or digging into the EVE runtime was performed, simply educated guesses and a knowledge of CPython internals. Progress is good. Relevant data identified. More results to be posted, and then an open source code release.

Update 1

Progress is good. The parser doesn’t yet understand type 0x2A yet, but it will soon… The parser library is a plain-vanilla C++ library, cross platform and embeddable in just about anything. I’m mainly focused on building an AST of the cache file format at this point. We can walk the AST and fetch the needed data later, which is the “easy part.” This is not a “find-magic-byte sequence and use fixed offsets” type parser :-) You can track progress at gitorious.org/libevecache.

Parse exception Can't identify
type 0x2a at position 0x124 Beginning dump...
 {Length: 3}   <SInt '738197504'>    <STuple>  
      (  {Length: 4}   <SMarker ID: -120 >    <SIdent 'GetOrders'>    <SInt '10000016'>    <SInt '642'>   ) 
  <SDict>  
      (  {Length: 1}   <SObject>  
      (  {Length: 2}   <SIdent 'objectCaching.CachedMethodCallResult'>    <STuple>  
      (  {Length: 2}   <SDict>  
      (  {Length: 4}   <STuple>  
      (  {Length: 3}   <SIdent '1 minute'>    <NONE>    <NONE>   ) 
  <SIdent 'versionCheck'>    <SMarker ID: -114 >    <SIdent 'sessionInfo'>   ) 
  <SSubstream>  
      (  {Length: 3}   <SLongLong '2678554370227830784'>    <SString 'dbutil.RowList'>    <SDict>  
      (  {Length: 4}   <SDBHeader>  
      (  {Length: 1}   <STuple>  
      (  {Length: 2}   <SString 'blue.DBRowDescriptor'>    <STuple>  
      (  {Length: 1}   <STuple>  
      (  {Length: e}   <STuple>  
      (  {Length: 2}   <SMarker ID: -117 >    <SInt '6'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: -95 >    <SInt '5'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: 74 >    <SInt '2'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: -116 >    <SInt '2'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: -118 >    <SInt '3'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: -96 >    <SInt '3'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: -119 >    <SInt '3'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: 116 >    <SInt '11'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: -125 >    <SInt '64'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: 126 >    <SInt '2'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: -101 >    <SInt '3'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: -115 >    <SInt '3'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: -106 >    <SInt '3'>   ) 
  <STuple>  
      (  {Length: 2}   <SMarker ID: 41 >    <SInt '3'>   ) 
 ) 
 ) 
 ) 
 ) 
  <SMarker ID: 36 >    <STuple>  
      (  {Length: e}   <SMarker ID: -117 >    <SMarker ID: -95 >    <SMarker ID: 74 >    <SMarker ID: -116 >    <SMarker ID: -118 >    <SMarker ID: -96 >    <SMarker ID: -119 >    <SMarker ID: 116 >    <SMarker ID: -125 >    <SMarker ID: 126 >    <SMarker ID: -101 >    <SMarker ID: -115 >    <SMarker ID: -106 >    <SMarker ID: 41 >   ) 
  <SIdent 'columns'>   ) 
 ) 
 ) 
 ) 
 )

Update 2

Success! I give you the first half of a market order cache file, including RLE decompressed database rows. The structure is not 100% correct in regards to nested elements, and there is a bit of unresolved magic occurring at the substream end condition. Also, the naming of the elements is somewhat odd - the first set of orders is actually the Sell orders, and the second half comprises the Buy orders. The exact meaning of the header defining the structure is also not yet determined (suspect its an identifier followed by a numerical “type” code, though either the order differs from the row descriptor or the types have no relation to the serialized types in the cache file…). The second half of the file can be parsed (except for the odd end condition again…), but the nesting structure is “broken”. This will require some more investigation, but is likely easy to fix. Another oddity is the dictionary type (or double tuple - the length is half of the number of elements present). It is encoded value followed by key (not key followed by value). Below is the AST structure dump from the ‘dumper’ utility included. The cache files list Armageddons in Lonetrek as of a couple days ago.

<SInt '738197504'>  
 <STuple>  
 (
   <SMarker ID: -120 >  
   <SIdent 'GetOrders'>  
   <SInt '10000016'>  
   <SInt '643'>  
 )
 <SDict>  
 (
   <SObject>  
   (
     <SIdent 'objectCaching.CachedMethodCallResult'>  
     <STuple>  
     (
       <SDict>  
       (
         <STuple>  
         (
           <SIdent '1 minute'>  
           <NONE>  
           <NONE>  
         )
         <SIdent 'versionCheck'>  
         <SMarker ID: -114 >

     <SIdent 'sessionInfo'>  
   )
   <SSubstream>  
   (
     <SLongLong '2678554370227830784'>  
     <SString 'dbutil.RowList'>  
     <SDict>  
     (
       <SDBHeader>  
       (
         <STuple>  
         (
           <SString 'blue.DBRowDescriptor'>  
           <STuple>  
           (
             <STuple>  
             (
               <STuple>  
               (
                 <SMarker ID: -117 >  
                 <SInt '6'>  
               )
               <STuple>  
               (
                 <SMarker ID: -95 >  
                 <SInt '5'>  
               )
               <STuple>  
               (
                 <SMarker ID: 74 >  
                 <SInt '2'>  
               )
               <STuple>  
               (
                 <SMarker ID: -116 >  
                 <SInt '2'>  
               )
               <STuple>  
               (
                 <SMarker ID: -118 >  
                 <SInt '3'>  
               )
               <STuple>  
               (
                 <SMarker ID: -96 >  
                 <SInt '3'>  
               )
               <STuple>  
               (
                 <SMarker ID: -119 >  
                 <SInt '3'>  
               )
               <STuple>  
               (
                 <SMarker ID: 116 >  
                 <SInt '11'>  
               )
               <STuple>  
               (
                 <SMarker ID: -125 >  
                 <SInt '64'>  
               )
               <STuple>  
               (
                 <SMarker ID: 126 >  
                 <SInt '2'>  
               )
               <STuple>  
               (
                 <SMarker ID: -101 >  
                 <SInt '3'>  
               )
               <STuple>  
               (
                 <SMarker ID: -115 >  
                 <SInt '3'>  
               )
               <STuple>  
               (
                 <SMarker ID: -106 >  
                 <SInt '3'>  
               )
               <STuple>  
               (
                 <SMarker ID: 41 >  
                 <SInt '3'>  
               )
             )
           )
         )
       )
       <SMarker ID: 36 >  
       <STuple>  
       (
         <SMarker ID: -117 >  
         <SMarker ID: -95 >  
         <SMarker ID: 74 >  
         <SMarker ID: -116 >  
         <SMarker ID: -118 >  
         <SMarker ID: -96 >  
         <SMarker ID: -119 >  
         <SMarker ID: 116 >  
         <SMarker ID: -125 >  
         <SMarker ID: 126 >  
         <SMarker ID: -101 >  
         <SMarker ID: -115 >  
         <SMarker ID: -106 >  
         <SMarker ID: 41 >  
       )
       <SIdent 'columns'>  
     )
     <DBRow 001882ba7d000000000000000000f03f509bcdfd85fcc90178a1af460500000001000000ac96930390969800d3c8c901050000008302ff7f5a> 
     <DBRow 001882ba7d000000000000000000144050a0f97765fec901a105c54608000000010000007390930390969800d3c8c901050000008302ff7f5a> 
     <DBRow f0c6933881000000000000000000f03f909da9a71a0eca019b048e480100000001000000468a930390969800d7c8c901010000008302ff7f5a000000> 
     <DBRow 004ee77e7d000000000000000000f03f9067651cec00ca013b9b434701000000010000007192930390969800eec8c901070000008302ff7f5a000000> 
     <DBRow 003476667b000000000000000000f03f300ef4eb3409ca0173a311480100000001000000c2bf930390969800eec8c901070000008302ff7f5a> 
     <DBRow 001882ba7d000000000000000000f03f50b65d2fc40eca0143029f480100000001000000149b930390969800f2c8c901040000008302ff7f5a> 
     <DBRow 001882ba7d000000000000000000f03f3024c3b6fc0eca01fe10a6480100000001000000149b930390969800f2c8c901040000008302ff7f5a> 
     <DBRow 00e09962820000000000000000001440b0a9347fe3dec901ec889b3c0a00000001000000179b930390969800f3c8c901050000008302ff7f5a> 
     <DBRow 00a8b10a870000000000000000000040e0883e7431e2c901039148440b000000010000000490930390969800f9c8c901070000008302ff7f5a> 
     <DBRow 9ca7b10a8700000000000000000008404015505adcfdc9012b53f94606000000010000002296930390969800f9c8c901070000008302ff7f5a000000> 
     <DBRow 8049016282000000000000000000f03f10c1e8bc0409ca01e9ad0e4801000000010000001996930390969800fbc8c901080000008302ff7f5a000000> 
     <DBRow 00f94a8f7c0000000000000000001440c0a05632d10fca01fd20bb4805000000010000003796930390969800fdc8c901060000008302ff7f1e> 
     <DBRow 809ddd657b000000000000000000084010aee06a2e10ca010c32c34803000000010000003796930390969800fdc8c901060000008302ff7f1e000000> 
     <DBRow 7068e38f7c000000000000000000f03f20ffe8eced0aca012b523b4802000000010000003196930390969800fdc8c901060000008302ff7f5a000000> 
     <DBRow 809ddd657b000000000000000000f03f1073931cab0fca01bda1354805000000010000003796930390969800fdc8c901060000008302ff7f1e000000> 
     <DBRow 00267c907c000000000000000000f03fb0a3e34d050eca01c4d68a4801000000010000002e9693039096980001c9c901080000008302ff7f5a> 
     <DBRow 0070c9b28b000000000000000000f03f50dfb0e8010fca0141d0a64801000000010000009d969303909698000fc9c9010a0000008302ff7f5a> 
     <DBRow 8049016282000000000000000000f03f40a6e8b2240bca013434424801000000010000006f9793039096980010c9c901090000008302ff7f5a000000> 
     <DBRow f0902e7481000000000000000000f03fb0cfaa9f370fca017790ac480100000001000000639793039096980012c9c9010b0000008302ff7f5a000000> 
     <DBRow 00a8b10a87000000000000000000f03fe0277593410fca01bd61ad480200000001000000169093039096980015c9c9010e0000008302ff7f5a> 
     <DBRow 00a8b10a87000000000000000000004000cbd3a70010ca0134c5bf480200000001000000169093039096980015c9c9010e0000008302ff7f5a000000> 
     <DBRow 001882ba7d000000000000000000f03fb0c2eda85110ca01d44dc5480100000001000000149b930390969800f2c8c901000000008302ff7f5a> 
     <DBRow f01373d17a000000000000000000f03fe0ea9be87910ca014d90c9480100000001000000fe95930390969800e1c8c901000000008302ff7f1e000000> 
     <DBRow 50a3b10a87000000000000000000f03fb0dae8bfa410ca013d22cf480200000001000000169093039096980015c9c901000000008302ff7f5a000000> 
     <DBRow 50a3b10a870000000000000000000040b0d4aadeab10ca01a8d9cf4802000000010000002296930390969800f9c8c901000000008302ff7f5a000000> 
     <DBRecords>  
     (
       <STuple>  
       (
         <STuple>  
         (
           <SStreamIdent 2>  
         )
         <SDict>  
         (
           <SStreamIdent 1>  
           <SMarker ID: 36 >  
           <STuple>  
           (
             <SMarker ID: -117 >  
             <SMarker ID: -95 >  
             <SMarker ID: 74 >  
             <SMarker ID: -116 >  
             <SMarker ID: -118 >  
             <SMarker ID: -96 >  
             <SMarker ID: -119 >  
             <SMarker ID: 116 >  
             <SMarker ID: -125 >  
             <SMarker ID: 126 >  
             <SMarker ID: -101 >  
             <SMarker ID: -115 >  
             <SMarker ID: -106 >  
             <SMarker ID: 41 >  
           )
           <SStreamIdent 3>  
         )
       )
     )
     <DBRow 005c3e736a0000000000000000001440d062bb918efdc9016f29f7450500000001000000bb96930390969800d8c8c90100000000830228005a0001> 
     <DBRow 9cbe2f2000000000000000000000f03ff0eb1ad2fefdc9017c43fc460100000001000000c397930390969800d8c8c901000000008302ff7f5a000100> 
     <DBRow 0427f5536d000000000000000000f03fb0cbb36f250bca015048424801000000010000000196930390969800e0c8c90100000000830203005a000100> 
     <DBRow 9cd50e2000000000000000000000f03f7023fa4b21dac9010b816c4301000000010000003196930390969800fdc8c901000000008302ff7f5a000100> 
     <DBRow 00a82d336f000000000000000000004030551585dc0fca01f7a5314805000000010000003196930390969800fdc8c9010000000083020a005a0001> 
     <DBRow 4068fe3f6e0000000000000000001c4060feb17c410bca01a402a9470a000000010000002e9693039096980001c9c90100000000830214001e000100> 
     <DBRow 00ba95b96e000000000000000000f03f30366bc52f0eca01711940480100000001000000919693039096980011c9c90100000000830204005a0001> 
     <DBRow e0067fe0680000000000000000001040302ebdaeedf4c901a9d6f0440400000001000000609793039096980012c9c90100000000830205005a000100> 
     <DBRow c84cb55b6b000000000000000000f03f408f5a9adb02ca0121208a460100000001000000609793039096980012c9c90100000000830205005a000100> 
     <DBRow b085fde068000000000000000000f03ff056816f43fac9016eb19e4601000000010000006c9793039096980014c9c90100000000830205005a000100> 
     <DBRow 801bf4266c000000000000000000004090cd97f0cf05ca019d7ead470200000001000000169093039096980015c9c90100000000830203005a000100> 
     <DBRow 40fa51b96e000000000000000000004010707d136b0dca01d021a0460300000001000000e69593039096980023c9c90100000000830214005a000100> 
     <DBRow 001882ba7d000000000000000000f03f1026ef856e10ca01bb37c8480100000001000000fd93930390969800f3c8c901000000008302ffff5a0001> 
     <DBRow 00a446167200000000000000000000408024d1608e10ca01044fcc480200000001000000e293930390969800e0c8c901000000008302ffff070001> 
   )
   <SLongLong '128934085416010580'>>  
 )

)

© 2020 Yann Ramin