Butterfly Backup : differential backup seems to behave like full backup

[didier33170]

Description

I was successfully daily backing up a single directory tree using the 'differential' mode:
bb backup --computer Mint-Didier --destination /media/didier/MyDrive --mode differential --custom-data /media/didier/Disk_A/Data_A --user didier --type unix --retention 7 7 --skip-error --exclude EXCLUDED
As expected, after the first backup, the used space on 'MyDrive' is increasing very slowly, following the space took by the modified or new files.

I'm now intending to daily backup 3 directories located on 2 disks:
bb backup --computer Mint-Didier --destination /media/didier/MyDrive --mode differential --custom-data /media/didier/Disk_A/Data_A /media/didier/Disk_B/Data_B1 /media/didier/Disk_B/Data_B2 --user didier --type unix --retention 7 7 --skip-error --exclude EXCLUDED
In this configuration the used space is increasing daily of the full size of the directories backed up (ie. Data_A + Data_B1 + Data_B2). It is like if BB performed a full backup every day.

Steps to Reproduce

See the command above.

Expected Behaviour

For a differential backup the used space should increase only following the space took by the modified or new files.

Your Environment

• Butterfly Backup version used: 1.10.0
• Operating System and version: Linux Mint 21.2, kernel 5.15.0-91

Additional context

Thank you anyway for your efforts to propose this nice tool !

[MatteoGuadrini]

Hi @didier33170,
Unfortunately looking at the code I can't understand where the problem is. The --custom-data argument accepts multiple paths at the same time and I compose the rsync command using the paths I receive; you should find this command if you enable --verbose or --log:

rsync -ahu --no-links --link-dest='last_full_backup' --exclude=EXCLUDED Mint-Didier@didier:/media/didier/Disk_A/Data_A :/media/didier/Disk_B/Data_B1 :/media/didier/Disk_B/Data_B2 /media/didier/MyDrive

Can you confirm this command for me? Can you maybe enable log or verbose mode?

Thank you so much for the compliments and thank you for using BB.

[didier33170]

Hi Matteo. --verbose was in the original command. However I deleted the log some days ago because I modified my strategy to avoid the problem. I'm now launching several instances of BB using crontab, one for each directory I want to backup.
This way the differential mode behave as expected. If necessary, I can temporarily return to the initial mode to get the rsync output.

[MatteoGuadrini]

I would be grateful if you could copy here the output of either the log or the verbose mode.
So I can investigate.
Thanks again

[didier33170]

Hi Matteo. Happy New Year! And sorry for the black-out of some weeks, due to a failure of the processor of my computer (AMD Ryzen 7).
I re-setup a cron command for a daily differential backup on 3 small directories (Thunderbird mails, Darktable database and Stellarium configuration), for a total of 2.7Go. It is to be noted that those sets of files do not change much during the 3 days of the test. Here are the results:
Total of occupied space on the backup disk = 8.2Go, that is close to 3 x 2.7Go (ie. like 3 x full backups).
Here is the BB command line in crontab:
0 13 * * * bb --verbose backup --computer Linux-Didier --destination /media/didier/Didier_2To/bb0 --mode differential --custom-data /home/didier/.var/app/org.darktable.Darktable/config/darktable /home/didier/.thunderbird/ytlv3xxx.default-release /home/didier/.stellarium/modules/Oculars --user didier --type unix --retention 7 7 --skip-error --exclude EXCLUDED >>/media/didier/Didier_2To/journal_butterfly_backup && echo "Fin sauvegarde : $(date)" >>/media/didier/Didier_2To/journal_butterfly_backup
... it also illustrate the purpose of my other request to store the folders address in a text file!
And here is the output of the verbose command:
success: Command rsync -ahu --no-links --link-dest=/media/didier/Didier_2To/bb0/Linux-Didier/2024_01_05__15_44 --quiet --exclude=EXCLUDED didier@Linux-Didier:'/home/didier/.var/app/org.darktable.Darktable/config/darktable' :'/home/didier/.thunderbird/ytlv3xxx.default-release' :'/home/didier/.stellarium/modules/Oculars' /media/didier/Didier_2To/bb0/Linux-Didier/2024_01_07__13_00
I hope that it can help you to understand if I did a mistake or if their is some other thing wrong.
Best regards

[MatteoGuadrini]

Hi @didier33170, I finally managed to reproduce the case study, but without unfortunately running into the bug you encountered with the differential mode.
This is command: bb backup -c RB1 --destination /tmp --custom-data /home/user1 /home/user2 --type unix --skip-error --mode differential

I launched it three times at different times; I started with an initial situation where the two users have more or less 9GB of data:

1. 12:42 PM: user1 and user2 they weigh 8.6GB
2. 12:47 PM: I add a Test directory with some data that weigh 2.7GB
3. 14:33 PM: I remove the Test directory; some other system files are changed at this point, like browser data, which weigh approximately 1GB

This is result:

[myuser@RBB_SRV]# du -sh /tmp/RB1/2024_02_02__12_42  #--> full
8.6G	/tmp/RB1/2024_02_02__12_42
[myuser@RBB_SRV]# du -sh /tmp/RB1/2024_02_02__12_57  #--> diff
2.7G	/tmp/RB1/2024_02_02__12_57
[myuser@RBB_SRV]# du -sh /tmp/RB1/2024_02_02__14_33  #--> diff
1.1G	/tmp/RB1/2024_02_02__14_33

[myuser@RBB_SRV]# ll -i /tmp/RB1/2024_02_02__12_42/user1/Desktop/ /tmp/RB1/2024_02_02__12_57/user1/Desktop/ /tmp/RB1/2024_02_02__14_33/user1/Desktop/
/tmp/RB1/2024_02_02__12_42/user1/Desktop/:
total 8
134932 -rw-r--r-- 3 502 games 1000 Jan 31 09:31 'Books.xlsx'
134933 -rw-r--r-- 3 502 games 1016 Dec  4 16:47 'Cars.xlsx'

/tmp/RB1/2024_02_02__12_57/user1/Desktop/:
total 12
134932 -rw-r--r-- 3 502 games 1000 Jan 31 09:31 'Books.xlsx'
134933 -rw-r--r-- 3 502 games 1016 Dec  4 16:47 'Cars.xlsx'
154747 drwxr-xr-x 6 502 games 4096 Feb  2 12:57 'Test'

/tmp/RB1/2024_02_02__14_33/user1/Desktop/:
total 8
134932 -rw-r--r-- 3 502 games 1000 Jan 31 09:31 'Books.xlsx'
134933 -rw-r--r-- 3 502 games 1016 Dec  4 16:47 'Cars.xlsx'

Note the inodes of the files on the Desktop that have not changed, they are identical because they are hard links in the Linux file system (in this specific case, I used fedora iot).

I don't know what else to investigate because from the results of the test done, everything seems fine to me.
At this point, before closing the issue without a real resolution, I would ask you to carry out further tests, analyzing the rsync backup log located in the folder of each backup performed: i.e /media/didier/Didier_2To/bb0/Linux-Didier/2024_01_05__15_44/backup.log