[Neo4j] Help us make Neo4j better at handling YOUR data

Tobias Ivarsson tobias.ivarsson at neotechnology.com
Fri Feb 4 17:13:13 CET 2011


Friends,

Please read this, your involvement will make Neo4j more efficient!

A week ago I spiked an idea for making storing short string properties more
efficient.

Since then me and my colleagues at Neo Technology have done some testings,
and the initial measurements are looking very promising.
We have measured string store sizes decreasing by up to 80%. With the added
benefit of those 80% of the string properties being accessible much faster.

In order to make these improvements applicable to as many stores as possible
I would like to get some statistics on what the strings stored by actual
applications look like.
For these statistics to be as accurate as possible I need your help.

I have written a small utility that analyzes the string properties stored by
Neo4j and computes some statistics about them.
If I could get as many of you to run this tool on your stores and send those
statistics to me as possible, that would be great.

This tool is available for download here:
https://github.com/downloads/thobe/neo4j-admin-store/stringstat.jar

To run it, all you need to do is:
java -jar stringstat.jar /path/to/your/neo4j/store/dir

Then send the output (printed to stdout at the end of execution) in
the STRING STORE STATISTICS block to me via e-mail.

Please keep in mind that even if my tests show that running this tool is
safe and does not modify your datastore, running it will be on your own
risk.

Since this tool only collects statistics about your string data, and not the
actual data, none of your sensitive information will be leaked by sharing
the statistics from this tool.

Once again, please consider running this tool and submitting the statistics,
since doing so will make Neo4j better at handling YOUR data in the future.

The rest of this email is a technical description of what the tool does and
the statistics it gathers.

This tool will first iterate through all string property records (filtering
out any record that occupies more than one block, since those are too long
to be applicable anyway) and compute the character frequencies in these
strings (under the assumption of three different compression levels).
It will then use this to design three custom compression formats that are
optimized for the short strings in your database.
These compression formats will be used together with six formats that would
be useful based on most databases I've seen to compute how large the gain of
compression on your store would be.
The resulting statistics on how efficient each compression format would be
is then written to standard out together with the character frequencies from
step one.

Since this process can take some time on a large store, I've added a
progress indicator that prints out while the tool is doing its thing. This
will print a dot for every half percent of progress and print how far
through the store it has gotten (in percent) every ten percent. This
progress indication will be printed twice since the tool iterates through
all string records twice, first to compute the frequencies, and then to
compute the gain of compression.


Here is an example of what the output from running this tool could look
like:

$ java -jar target/stringstat.jar /Users/tobias/tmp/neo4j-db
Computing character frequencies for 12395202 string records
................... 10%
................... 20%
................... 30%
................... 40%
................... 50%
................... 60%
................... 70%
................... 80%
................... 90%
...................100%
Matching potential encodings for 12395202 string records
................... 10%
................... 20%
................... 30%
................... 40%
................... 50%
................... 60%
................... 70%
................... 80%
................... 90%
...................100%
<STRING STORE STATISTICS>
= 4 bit frequencies =
  1: 1327696  2: 1248529  5: 1248296  8: 1247808
  4: 1247263  0: 1246971  3: 1246731  6: 1246669
  7: 1246401  9:  911489  a:  172562  n:  124640
  e:  123376   :   65844  m:   51293  g:   50802
  t:   50388  i:   49993  r:   44902  s:   33587
  k:   27985  b:   27031  .:   22444  o:   19244
  l:   15134  u:   10314  d:    9930  j:    9083
  S:    8593  L:    7706  M:    7564  A:    7258
= 5 bit frequencies =
  a: 268274  n: 267067  e: 256386   : 210979
  i: 196894  r: 154346  l: 134354  o: 131955
  s: 115836  d:  93797  t:  76991  k:  63191
  u:  60566  m:  57359  g:  53697  S:  51287
  .:  50007  @:  49330  v:  46931  h:  42891
  M:  42399  H:  37493  A:  34785  L:  31556
   j:  31315  b:  29391  E:  28335  y:  26427
  T:  26248  B:  24491  c:  23347  K:  22046
  f:  20345  p:  20287  J:  20173  N:  19144
  ?:  16926  W:  16001  G:  14017  R:  13871
  O:  13630  F:  13054  D:  12195  V:  11483
  z:  10023  ?:   8814  ?:   8702  ?:   7903
  x:   6439  U:   5837  C:   5371  I:   4547
  ?:   4491  2:   2703  P:   2637  4:   1218
  w:   1187  Z:   1183  Y:   1178  ,:   1141
   -:    857  ?:    634  1:    632  0:    316
= 6 bit frequencies =
  n: 874452  a: 802639  e: 790998  i: 597945
   : 524600  r: 496695  o: 462202  s: 457249
  l: 429857  d: 327564  t: 310925  .: 217419
  m: 207459  @: 203896  k: 190190  g: 180166
  u: 169502  v: 158119  S: 148109  h: 144968
  M: 118409  H: 101946  b:  93958  c:  85548
  A:  84580  E:  77119  L:  75604  y:  67204
   j:  66711  K:  63279  T:  61919  p:  60935
  f:  60589  B:  51351  J:  48471  ?:  44154
  N:  39113  O:  35251  W:  32999  F:  32734
  G:  30821  R:  30341  z:  27002  V:  25837
  ?:  25434  D:  23706  C:  22801  I:  21277
  ?:  19167  ?:  17891  ?:  14752  P:  13294
  x:  11184  2:  11011  w:   9972  U:   9556
  Y:   7486  -:   4537  Z:   4524  ?:   4496
  1:   4490  4:   3119  q:   1756  0:   1379
   6:   1379  9:   1379  8:   1379  ,:    609
  ?:    342  ?:    294
   9107508 strings with category bitmask              0b0
   2947442 strings with category bitmask        0b1111111
     80361 strings with category bitmask       0b10000000
      8251 strings with category bitmask      0b100000000
     16971 strings with category bitmask      0b100010000
     26235 strings with category bitmask      0b100010001
     20632 strings with category bitmask      0b100010011
     41383 strings with category bitmask      0b101010011
     86358 strings with category bitmask      0b101111111
       508 strings with category bitmask      0b110000000
     32973 strings with category bitmask      0b110010000
     26579 strings with category bitmask      0b110010001
= Category index =
 0. NineSevenBitAscii
 1. LowerCaseHexadecimal
 2. UpperCaseHexadecimal
 3. PunctuatedNumerical
 4. AlphaNumericalName
 5. Numerical
 6. FrequencyBased:4bit
 7. FrequencyBased:5bit
 8. FrequencyBased:6bit
</STRING STORE STATISTICS>


The information I want you to send back to me is the data between
the <STRING STORE STATISTICS> and </STRING STORE STATISTICS> lines.

May the source be with you,
-- 
Tobias Ivarsson <tobias.ivarsson at neotechnology.com>
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857


More information about the User mailing list