Bugs in couchdb-dump-php!

[tomas_]

Hi!
I don't know if this is the right forum, but I'll try anyway.

A while ago I noticed some bugs in the couchdb-dump-php tools (https://github.com/2600hz/couchdb-dump-php).
I didn't see the impact of this until today, when accidentally removed my company's whole database in Couch(!) (oops, took 3 seconds with a faulty curl command, and 6-7 hours to restore from backup).

We're using couchdb-dump.php to make nightly backups of the whole CouchDB (https://forums.2600hz.com/forums/topic/9275-couchdb-backup-strategies/?do=findComment&comment=51413)
However, it seems like the restore script imports the objects as strings instead!
In devices documents, ' "caller_id": { }, '' gets imported as ' "caller_id": [ ], ', same with contact_list etc and also in users documents. To solve this you can use the sed commands below, in the backup file before restoring it.

sed -i -e 's/caller_id\":\[\]/caller_id\":\{\}/g' /tmp/backup.json
sed -i -e 's/contact_list\":\[\]/contact_list\":\{\}/g' /tmp/backup.json
sed -i -e 's/dial_plan\":\[\]/dial_plan\":\{\}/g' /tmp/backup.json
sed -i -e 's/contact_list\":\[\]/contact_list\":\{\}/g' /tmp/backup.json
sed -i -e 's/music_on_hold\":\[\]/music_on_hold\":\{\}/g' /tmp/backup.json
sed -i -e 's/ringtones\":\[\]/ringtones\":\{\}/g' /tmp/backup.json
sed -i -e 's/call_restriction\":\[\]/call_restriction\":\{\}/g' /tmp/backup.json
sed -i -e 's/profile\":\[\]/profile\":\{\}/g' /tmp/backup.json
sed -i -e 's/media\":\[\]/media\":\{\}/g' /tmp/backup.json

Another thing that doesn't work as expected are the phone numbers. The plus sign is imported into space instead;

To solve this you need to copy, then delete and create a new number doc and replace space with plus, then save. On every number, manually...


We tried to look into the code of the scripts, but can't see why this bugs occur. Hopefully some more experienced developers out there can fix this!

Edited April 26, 2018 by tomas_ (see edit history)

[tomas_]

Hi!
Today it happened again, this time with bigger impact - Many databases discarded when an old, removed couchdb node (with old databases) went online after a power outage and somehow synced the old database with the other nodes. (Not good at all, you learn something each day! ).
(Since time we've started to make tar backups of the /srv folders on each db node also, but the last few days the disks went full and our automatic VPS script to increase disk storage wasn't working. Bad luck!!)

So we started to restore databases which was backed up by our script running couchdb-dump, but then we ran into the issues in my post above!
The number docs was easily fixed, just replace the + sign in the number/doc id with %2B. Sed is a really good tool, to fix we just ran:

sed -i -e 's/_id":"+/_id":"%2B/g' number*

 

However, the issues still remains. Anyone that can see why?
There must be other out there using the couchdb-dump tool which has the same issues?

Br - Tomas