Backups and the new Jet backend

[sugoi]

Hi.

I saw that Plastic 6 has an all new, super fast backend named Jet and got really curious about it.

I didn't see anything mentioned about backups, so I decided to ask how it will work.

 

Do you have to stop the server and copy the files like SQLite or is there support for backing the system up like SQL Server, where you can backup without stopping the server.

 

Thanks!

[psantosl]

Hi Sugoi,

 

Thanks for reaching us! We are very excited about Jet, it is really much faster than anything else.

 

Regarding the backup:

* At this point you need to stop the server to do a backup. We have been testing it with rdiff on Linux, doing incremental backups, and it works great. We will publish official docu soon. @manu has more info about it since he is doing the tests.

* Our goal is to have a command to put the server in "readonly mode", so you can do the backup without stopping. While the backup is in progress (incremental should be fast) the server will continue allowing pulls, and updates, and browsing and diffing, but will deny pushes, creating branches, checkins... Normally this will be good for most of the setups.

* You can use a secondary server and replicate to it to have a backup too => this can be an option for 24x7 servers.

* Finally, we have studied the option to develop our own full always-on backup: but then we can't rely on any existing backup tech, we have to create our own dumps while the server is running. Not super hard, but we didn't start the effort yet, although some of our big customer (+1000 developers) are already interested on it. Most of them are pointing to option 3 as a valid solution.

 

Cheers,

 

pablo

[sugoi]

Very interesting answer.

I really like the idea of putting the server in readonly mode, it would work really well.

The replication scenario isn't as interesting for a smaller shop, since it would incur maintenance costs (extra server or cloud) and introduce extra steps:

* setup automatic replication

* detect when replication is done, then stop the secondary server to put its DB into consistent state

* create the backup

* spin up the secondary server

But for this to be more reliable, we would have to also set monitoring of the secondary server, and failure recovery as well, so the first server won't go crazy when automatic replication kicks in the the secondary is dead. We also need to be careful not to stop the secondary during a replication.

 

With the readonly option it would be very simple to perform the backups using the same setup most of us already have (setups that stop the server before the backup operation, then start the server again).

 

Of course, a in-house backup solution would be even better, as it would have access to Plastic's internals and could speed up backups immensely. One thing that comes to mind is the timestamps in SQLite DBs. Restarting the Plastic server changes the timestamps and the DBs are seem as changed by the backup tools, which, in turn, wastes time as the tool will have to check the file to see what exactly changed (for incremental/differential backups, the entire file will need to be read).

 

 

Thanks for such a detailed answer.  Really looking forward to seeing how Jet will turn out!

 

P.S.: Incidentally, are you planning on enabling Jet on community/open source licenses as well, or only for team/enterprise licenses?

 

 

 

[psantosl]

Jet will be available for everyone, including free licenses. That's the plan.

 

Of course, big enterprise setups will benefit from the Enterprise Edition engine improvements, different access patterns and so on.

But yes, we would like to include Jet as the default replacing SQLite, indeed.

[sugoi]

Hey, @manu, @psantos, any update on the "readonly mode" command for the server?

[manu]

Hi!

We think it will be ready during the summer.Two month or so.

[cmtodd]

Re: option to replicate to another server and then do backups from there...

Is there a single command that can be run to replicate ALL repos from another server to this one (backup server)?  Also create repos if they do not exist?  guess I'm asking for a special replication command designed specifically for backup.

You see, we have A LOT of repos (1200+ and growing everyday).  Not that we couldn't create our own system to mimic the repos on the production server and setup a sync replication on the backup server for each of the repos, but a built in feature to do this would be much better.

Would REALLY like to look seriously at using JET but need a 24/7 uptime solution.  The "readonly" solution could work for us, but only if it might take 2-3 min for the sync while in readonly mode.  Anything longer would not be considered acceptable.  Better yet would be a continuous type of mirror we get today by setting up a slave MySQL server.

 

Thanks

[sugoi]

On 6/20/2017 at 2:57 PM, cmtodd said:

Re: option to replicate to another server and then do backups from there...

Is there a single command that can be run to replicate ALL repos from another server to this one (backup server)?  Also create repos if they do not exist?  guess I'm asking for a special replication command designed specifically for backup.

You see, we have A LOT of repos (1200+ and growing everyday).  Not that we couldn't create our own system to mimic the repos on the production server and setup a sync replication on the backup server for each of the repos, but a built in feature to do this would be much better.

Would REALLY like to look seriously at using JET but need a 24/7 uptime solution.  The "readonly" solution could work for us, but only if it might take 2-3 min for the sync while in readonly mode.  Anything longer would not be considered acceptable.  Better yet would be a continuous type of mirror we get today by setting up a slave MySQL server.

 

Thanks

 

Using dedicated drives for Plastic and a good incremental block-based backup (like Macrium Reflect) might fit your needs. I only have a very small number of repos and all are small, but I use Macrium to backup my OS drive and an incremental backup is very fast.

 

Since readonly mode was released, I wrote a few powershell helpers to put plastic in readonly mode, run the backup and put plastic back in normal model.

It uses MailAlert.exe to send me an email if something went wrong. I'll paste it below for future travelers.

 

 

function Main() {
    $mailAlert = 't:\MailAlert\MailAlert.exe';
    $errorMailSubject = "Error with PLASTIC backup";
    RunPlasticCommand -Command 'admin readonly enter' -FailureMessage 'Failed to run pre-backup command';
    ExitIfWrongReadonlyStatus -Active;

    #run your backup here

    RunPlasticCommand -Command 'admin readonly leave' -FailureMessage 'Failed to run post-backup command';
    ExitIfWrongReadonlyStatus -Inactive;
}

function RunPlasticCommand {
    Param(
        [Parameter(Mandatory=$True)]
        [String]$Command,
        [String]$FailureMessage
    )

    $psi = New-object System.Diagnostics.ProcessStartInfo;
    #$psi.CreateNoWindow = $true;
    $psi.UseShellExecute = $false;
    $psi.RedirectStandardOutput = $true;
    $psi.RedirectStandardError = $true;
    $psi.FileName = 'C:\Program Files\PlasticSCM5\client\cm.exe';
    $psi.Arguments = $Command;
    $process = New-Object System.Diagnostics.Process;
    $process.StartInfo = $psi;
    [void]$process.Start();
    $stdout = $process.StandardOutput.ReadToEnd();
    $stderr = $process.StandardError.ReadToEnd();
    $didExit = $process.WaitForExit(2 * 60000); # 2 minutes timeout, change according to your server

    if (!$didExit) {
        $process.Kill();
        ExitWithTimeoutMessage;
    }

    $iCmExitCode = $process.ExitCode;
    if ($iCmExitCode -ne 0) {
        $errorMail = "$FailureMessage. Exit code $iCmExitCode. stdout: $stdout stderr: $stderr";
        Start-Process -FilePath $mailAlert -ArgumentList ("-s ""$errorMailSubject"" -b ""$errorMail""");
        DisableReadonlyMode;
        Exit 1; # 1=backup error
    }

  return $process.ExitCode, $stdout, $stderr;
}


function ExitWithTimeoutMessage {
    $errorMail = "$FailureMessage. Timeout waiting for commmand to complete. Command: $Command";
    Start-Process -FilePath $mailAlert -ArgumentList ("-s ""$errorMailSubject"" -b ""$errorMail""");
    DisableReadonlyMode;
    Exit 1; # 1=backup error
}


######################################################
#     WILL DISABLE READONLY MODE BEFORE EXITING
######################################################
function ExitIfWrongReadonlyStatus {
    Param( 
        [Switch]$Active,
        [Switch]$Inactive
    )


    $iExitCode, $StdOut, $StdErr =  RunPlasticCommand -Command 'admin readonly status'
    $StringToFind = if ($Active) {"Active"} elseif ($Inactive) {"Inactive"};


    if (!($StdOut -like "*$StringToFind*")) {
        if ($Active) {
            DisableReadonlyMode;
        }

        $errorMail = "Fail. Plastic readonly mode is |$StdOut| but should be |$StringToFind|";
        Start-Process -FilePath $mailAlert -ArgumentList ("-s ""$errorMailSubject"" -b ""$errorMail""");
        Exit 1; # 1=backup error
    }
}

function DisableReadonlyMode {
    #if the server should be non-readonly, try to leave readonly mode before exiting
    RunPlasticCommand -Command 'admin readonly leave' -FailureMessage 'Failed to put server in non-readonly mode before exiting! The server will never leave readonly mode!';
}


#run the main function
Main;

 

[calbzam]

Wow, Thank you very much for sharing!

Regards,

Carlos.

[Shruti1605]

Hi Manu,

Could u please provide the clear instructions about installing rdiff-backup on linux machines and how to configure to use it for incremental backup. 

Is there any detailed document available for the same.

[calbzam]

Please check the followng link:

https://www.plasticscm.com/documentation/administration/plastic-scm-version-control-administrator-guide

Regards,

Carlos.