Having worked on several altcoin projects I can say one thing they all have in common is poor sync performance. Perhaps they were forked from Bitcoin before headers first sync was added, perhaps their code has been so hacked about that it's just too buggy. I often see obvious sync performance issues swept under the carpet with "just use the bootstrap". Today I wish to present a visualization tool called syncmon and a case study.
syncmon is very simple in concept, but very useful in application. It calls getinfo at minute intervals, writing time, connection count, header count and block count to a csv file which can be imported into your favorite spreadsheet or charting tool. This results in a chart showing the progress of header and block sync. Here is an example of a bugged sync process I worked on recently:
This sync was done with a single peer for clarity (using -connect command line option). What you can see happening here is that the headers sync is progressively slowing down until a node disconnect/reconnect forces the re-initialization of internal state data with the sync resuming at full speed, only to progressively slow again. Armed with this insight the dev team were able to determine that the syncing node was requesting progressively more redundant headers from its peer resulting in duplicate processing of headers already received. With this bug resolved, the sync time was reduced from 14 hours to 6 hours:
So the moral of the story is – collect stats and chart them. The shape of lines can tell you a lot more about what's going on, and better inform bug resolution. Next time someone tells you to "just use the bootstrap", you know what to do …