aws virtual tape library as storage for bareos
TRANSCRIPT
![Page 1: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/1.jpg)
AWS Virtual Tape Library as storage for Bareos
![Page 2: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/2.jpg)
The story behind the setup
Legacy Bacula (5.2) installation
Physical datacenter
HP MSL 2024: 24 slots, 1 tape drive, autochanger
MSL attached to the server, speed was good
It worked quite well, but…
![Page 3: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/3.jpg)
The story behind the setup
Prehistoric Bacula version: 3.0
![Page 4: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/4.jpg)
The story behind the setup
Prehistoric Bacula version: 3.0
…after upgrading from 2.4.
![Page 5: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/5.jpg)
The story behind the setup
Prehistoric Bacula version: 3.0
…after upgrading from 2.4
Old Red Hat OS.
![Page 6: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/6.jpg)
The story behind the setup
Prehistoric Bacula version: 3.0
…after upgrading from 2.4
Old Red Hat OS.
4.0
![Page 7: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/7.jpg)
More history
Once a week full backup tapes were vaulted off-site
![Page 8: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/8.jpg)
More history
Once a week full backup tapes were vaulted off-site
Sometimes backups did not finish on time
![Page 9: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/9.jpg)
More history
Once a week full backup tapes were vaulted off-site
Sometimes backups did not finish on time
Sometimes the tapes did not fit in the safe case
![Page 10: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/10.jpg)
More history
Once a week full backup tapes were vaulted off-site
Sometimes backups did not finish on time
Sometimes the tapes did not fit in the safe case
The whole tape pool did not fit in MSL + offsite
![Page 11: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/11.jpg)
Even more history (last slide!)
We wanted to upgrade the backup service
Zero risks allowed and no service interruption
![Page 12: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/12.jpg)
Even more history (last slide!)
We wanted to upgrade the backup service
Zero risks allowed and no service interruption
Duplicate infrastructure: not cheap
Start experimenting with AWS VTL
![Page 13: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/13.jpg)
What is AWS Storage Gateway-VTL?
Part of Storage Gateway service
Connect on-premises appliance with cloud storage
Virtual Tape Library / Volume gateway
On-premises VTL storage gateway acts as a cache
![Page 14: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/14.jpg)
What is AWS Storage Gateway-VTL?
![Page 15: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/15.jpg)
The appliance (VTL)
![Page 16: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/16.jpg)
VTL appliance
iSCSI devices exposed:
1 media changer
10 tape drives
1600 slots, 1600 import/export slots
I/E slots for archival/retrieval
![Page 17: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/17.jpg)
Cache Storage/Upload Buffer
At least two disks (DAS/SAN)
Cache Storage: Fast access to most recently used data
Upload buffer: in-transit data to S3 storage
We can add more disks later
![Page 18: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/18.jpg)
Cache Storage/Upload Buffer
Adapt LAN to Internet speed: size them properly:
Upload buffer at least 150G
Cache storage: 1.1 x upload buffer
YMMV: formula in the documentation
We can delete Upload Buffer (needs VTL restart)
![Page 19: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/19.jpg)
Cache disk deletion
We can delete Cache Storage disks
Officially not supported, but AWS support confirmed it was possible.
The cache must be reset, so be sure it is clean
CloudWatch metrics:
UploadBufferUsed
CachePercentDirty
![Page 20: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/20.jpg)
The Virtual Tapes
![Page 21: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/21.jpg)
Virtual Tapes
Analogous to the good old physical tape cartridges
From 100G to 2.5TB per tape
Backed in S3 storage
Up to 1500 tapes or 150TB of data per gateway
Managed via AWS web console or API
![Page 22: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/22.jpg)
The Virtual Tape Shelf
![Page 23: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/23.jpg)
Virtual Tape Shelf
Analogous to off-site tape holding facility
Stored in Amazon Glacier
Lower cost (4x times cheaper than S3)
24h retrieval time
One VTS per account per AWS region
![Page 24: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/24.jpg)
Virtual Tape Shelf
Archival via Import/Export slots in the VTL
Retrieval management via Web Console or API
Retrieved tapes visible in Import/Export slots
Read-only tapes after retrieval
![Page 25: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/25.jpg)
Read-only tapes?
WTF?
What about recycling?
![Page 26: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/26.jpg)
![Page 27: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/27.jpg)
Let me talk you about…
One of the cloud best-practices
![Page 28: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/28.jpg)
Embrace the constraints
![Page 29: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/29.jpg)
Let me talk you about…
One of the cloud best-practices
Embrace the constraints
Why do we love tapes?
![Page 30: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/30.jpg)
Let me talk you about…
One of the cloud best-practices
Embrace the constraints
Why do we love tapes?
We should love data, not the place where it is stored
![Page 31: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/31.jpg)
![Page 32: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/32.jpg)
Rethink Tape lifecycle
No Bareos-managed tape lifecycle (if we want archival)
BUT we can manage the storage gateway via API
Maybe a plugin could be written?
![Page 33: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/33.jpg)
Storage Gateway setup
Using the AWS Web console, service Storage Gateway
“Deploy new Storage Gateway”, “Gateway-Virtual Tape Library”
Download the VM template (ESX or HyperV)]
Download and deploy the on-premises appliance
Activate the gateway (register the appliance)
![Page 34: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/34.jpg)
Bareos server setup
Instal iscsi initiator tools (open-iscsi in Debian)
Discover the targets
iscsiadm --mode discovery --type sendtargets \--portal GATEWAY_IP:3260
![Page 35: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/35.jpg)
Bareos server setup
Login to each one of the targets
Increase iSCSI timeouts (VTL is uploading via Internet!)
iscsiadm --mode node \--targetname iqn.1997-05.com.amazon:sgw-852ecaec-tapedrive-01 \--portal GATEWAY_IP:3260,1 --login
![Page 36: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/36.jpg)
Your new devices
# lsscsi --generic
[9:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st0 /dev/sg2
[10:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st1 /dev/sg3
[11:0:0:0] mediumx STK L700 0103 /dev/sch0 /dev/sg4
[12:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st2 /dev/sg5
![Page 37: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/37.jpg)
Media changer: mtx shot first
Nice /dev/sch0 iSCSI changer device file
mtx won’t support it
From the man: “Control SCSI media changer devices”
/dev/sch0 is not an sg device, or old sg driver
![Page 38: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/38.jpg)
It’s okay to be generic
As mtx seems to be drunk, we need to use the generic device
/dev/sg4 to the rescue!
We can use that device file for Bareos configuration
![Page 39: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/39.jpg)
It’s okay to be generic
As mtx seems to be drunk, we need to use the generic device
/dev/sg4 to the rescue!
We can use that device file for Bareos configuration
Or can we?
![Page 40: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/40.jpg)
Media changer: Linux shot next
Modern kernels detect dynamically connected devices
Device numbers are assigned at boot time
![Page 41: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/41.jpg)
Media changer: Linux shot next
Modern kernels detect dynamically connected devices
Device numbers are assigned at boot time
VTL: 11 generic devices all connected at the same time
What can go wrong?
![Page 42: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/42.jpg)
Media changer: Linux shot next
Modern kernels detect dynamically connected devices
Device numbers are assigned at boot time
VTL: 11 generic devices all connected at the same time
What can go wrong?
“The autochanger device file changes” can go wrong
![Page 43: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/43.jpg)
Udev rules to the rescue!
Magic spell to assign always a known name to the changer
Save the spell in /etc/udev/rules.d/80-vtl-autochanger.rules
Use /dev/autochanger in Bareos config
SUBSYSTEM=="scsi_generic", SUBSYSTEMS=="scsi", ATTRS{type}=="8", \IMPORT{program}="scsi_id --sg-version=3 --export --whitelisted –d $devnode", \SYMLINK+="autochanger"
![Page 44: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/44.jpg)
What about tape drives?
Default udev rules are OK
![Page 45: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/45.jpg)
What about tape drives?
Default udev rules are OK
If you are OK with device names like
/dev/tape/by-path/ip-10.4.7.100:3260-iscsi-iqn.1997-
05.com.amazon:sgw-852ecaec-tapedrive-01-lun-0-nst
![Page 46: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/46.jpg)
Bareos configuration: Storage
Just use them like regular SCSI devices
Device {
Name = "tapedrive-0"
DeviceType = tape
DriveIndex = 0
ArchiveDevice = "/dev/tape/by-path/uninteresting.iscsi.stuff-tapedrive-01-lun-0-nst"
MediaType = ULT3580-TD5
AutoChanger = yes # default: no
}
![Page 47: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/47.jpg)
Bareos configuration: Storage
Autochanger {
Name = "VTL-autochanger"
Changer Device = /dev/autochanger
Device = tapedrive-0
Device = tapedrive-1
Changer Command = "/usr/lib/bareos/scripts/mtx-changer %c %o %S %a %d"
}
![Page 48: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/48.jpg)
Bareos configuration: Director
Storage {
Name = VTL
# Do not use "localhost" here
Address = ip-10-4-8-76 # YES, I CHEATED!
Password = "DummyPassword"
Device = "VTL-autochanger"
Media Type = ULT3580-TD5
}
![Page 49: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/49.jpg)
Bareos configuration: Director
Pool {
Name = VTLFull
Pool Type = Backup
Storage = VTL
[...]
}
Set up Jobs or JobDefs to use those pools
Or set up the storage in the Job resource
![Page 50: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/50.jpg)
Changer operation from bconsole
Very cool Bareos features!
Import and export (archive) tapes from bconsole itself
Even move tapes around the slots
Remember to update slots
status slots storage=VTL
![Page 51: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/51.jpg)
Questions?
![Page 52: AWS Virtual Tape Library as storage for Bareos](https://reader031.vdocuments.site/reader031/viewer/2022030320/586b59271a28ab6b518ba7f9/html5/thumbnails/52.jpg)
Barcelona
Plaça de Catalunya, 1
08002 - Barcelona
Madrid
Cerro de los Gamos, 1
28224 – Pozuelo de Alarcón
www.capside.com [email protected]
linkedin.com/company/capside @capside
Thank you!