1 October 2023 by Remco Bouckaert
Trace and tree logs can become quite large, especially with large numbers of taxa. Here are a number of ways to reduce the file sizes.
Set pre-burn-in on MCMC
Pre-burnin is like burn-in, except it does not log the samples. Be aware that for post-processing it is not necessary any more to remove burn-in. Most post-processing programs (like Tracer) assume a default of 10% burn-in, so you need to change these settings when analysing the logs.
The pre-burn-in can be set in the MCMC panel of BEAUti, or you can edit the XML in a text editor and add the
preBurnin attribute for the
logEvery so that the number of samples=10000
Every logger has a
logEvery attribute that determines how many MCMC samples are skipped before saving a state in the trace a tree logs.
Usually, it does not make sense to have more than 10 thousand samples in the log, since it will not result in better accuracy of the represented distributions in the trace log, assuming the chain mixed well.
By dividing the length of the MCMC chain by 10 thousand, you can calculate what
logEvery should be.
For example, for a chain length of 10 million, we get 10,000,000/10,000 = 1e7/1e4=1e3 = 1000, which are the default settings for the Standard template in BEAUti.
dp on TreeWithMetaDataLogger for tree loggers
dp attribute represents the number of decimal places to use writing branch lengths, rates and real-valued metadata.
If you use -1 full precision is used, and this is the default.
This means branch lengths and rates are logged in about 17 significant digits.
dp to something lower, e.g.
dp="5" the tree file would ouput
which is half the size.
To set the
dp, in BEAUti go to the MCMC panel, click on the triangle next to the tree log, and select the button with
TreeWithMetaDataLogger.t:partition next to it (where
partition the name of the partition). A dialog pops up where you can set the entry for
Alternatively, edit the XML in a text editor, and search for the
log element with
TreeWithMetaDataLogger in it. Add
dp="5" to set the number of significant digits to 5.
For example like so:
<log id="TreeWithMetaDataLogger.t:dna" dp="4" spec="beast.base.evolution.TreeWithMetaDataLogger" tree="@Tree.t:dna"/>
Use LogCombiner to thin existing log files
LogCombiner has a
-resample option that outputs only those entries at the given frequency.
For example, if you have a log that stored every 1000th sample, but you only need log entries every 10 thousand samples, you can call LogCombiner using
/path/to/logcombiner -resample 10000 -log input.log -o output.log
Alternatively, in the GUI there is a checkbox next to
Resample states at lower frequency that you can set, and then specify 10000 in the entry next to it.