Veeam hard disk read slow Search 5/29/2024 10:04:01 Veeam Community discussions and solutions for: Backup virtual machine with large volume backup server" or using the "file level backup " backup method (very slow. At the very least, the disk read speed has never been this slow in the past, even with incremental backups. 589 as a solution to back up our barebone MS SQL Server 2017 on Windows Server 2016, backing up whole server using installed VEEAM CBT driver. 5 . Top So I've a test file server, W2019 + IIS. Every time I try to have Veeam backup our SQL server, the disk latency for the SQL server increases within days after the backup. Any read write action would lead to 100% disk usage with barely a few KB of data transfer actually happening. Bottleneck: Target Disk read (live data, since job is running now) are ~4-5MB/s per/Harddisk. Jobs Backup Veeam Community discussions and solutions for: performance (very very slow speed) these backup has very slow performance (read at less than10 MB/sec as speed) investigating in statistic for the job one can see in the problematic virtual machine (with two hard disk) that one disk stay for hours at "59,9 GB read at 5 MB/sec" (Hard disk is Veeam Community discussions and solutions for: slow backup speed 02612135 of Veeam Agent for Microsoft Windows. That's my 2 cents at least. 5 hours. 04. 1 GB of data? It was registering speeds of <2 MB/s most of the time. Hard disk 1 (50. If you test with like DiskSpd, what are your results? Top. Using Storage Snapshots with a physical SAN-attached proxy would not really help here, since it just changes where the read I/Os happen. Is it the way it is supposed to proceed ? Below the message I have in statistics: Using target proxy X. Your direct line to Veeam R&D. Both servers are in the same rack, connected via two GBit switches. It seems only slow when the server has been in use - weekends it runs fine (about 30 mb/s read on VMs) but in the week this slows to 11 mb/s. I'm wondering if anyone else has seen extremely slow FLR mounts after upgrading from Veeam 10 to 11. Now when I resumed them the calculating digests step is taking forever, like 60 minutes for a 50GB hard drive, this single job would take days at this rate. If I recall correctly, I remove the disk from the VM, and then add During the most recent backup (incremental), the read speed was 174 MB/s on drive C and 83 MB/s in drive D, but for the current job in progress those speeds are 36 MB/s and 22 MB/s . 5 or an earlier version. One vdisk is made of 3 disks and runs in raid 5. 0 TB): 188. Potentially, any process that interacts with a backup file may hang when trying to open a backup file. Your speeds are quite slow, so to me, that seems like an infrastructure or config issue. Do you get Load results from a restore job? (e. I am verry close to just giving up,and let read whatever it wants Veeam Community discussions and solutions for: "Using backup proxy VMware Backup Proxy for disk Hard disk <number> [nbd] " So it's using NDB. Now for disk 6 and 7 it tells me: Preparing backup proxy NL-HQ-VEEAM01 for disk Hard disk 6 [hotadd] It seems like it is stuck, because it is only running on 2 of the 4 available threads. Each hard disk is 3 TB capactity and utilizing 1 backup thread for 1 Hard disk. Not a support forum! I was able to get clarification that each backup job will need to read the entire disk, even if the VM is part of other jobs that have completed. 0 GB) 42. CelticDubstep Influencer The AVHDX is a Veeam Community discussions and solutions for: Backup Incredible Slow - Processing at about 200kb/s of VMware vSphere R&D Forums. 5 hosts (a file server and Exchange server on one, an SQL server and the Veeam VM on the other) Production storage is a Synology DS1513+ with four WD Red drives configured in RAID 10. 3 ms Backend disk is 64KB ReFS on a R5 512Kb striped SSD array. How are these values? Host 2 has more than one network connection connected and Veeam is using the slower connection somehow. Mehnock Influencer Posts: 20 I've got an issue at the moment where backup jobs are taking a longt time to begin processing - they get to the "Hard disk 1 (0. and appears to have re-read the entire hard drive. Each server has a local SAS storage (Source RAID-10 and Target RAID-6). 6GB read at 8Mb/s [CBT] - 16:01:02 Thats 16 hours! This is blowing our backup window and causes backup-replication scheduling conflicts. Using file management capabilities Veeam Backup provides, copy some VMDK file (a few gigs) of non-running VM from your production ESX storage to Veeam Backup console hard disk. I also don't like having large backup chains open. x Btw, when you say 30MB/s, do you mean the entire job processing rate or the hard disk read speed? With NBD transport mode, processing is limited to around 30MB/s, if I recall correctly. I have 2 backup jobs that run nightly and have been going fine other than speed. The volume is hosted on a Compellent SC4020 attached via 10 Gb iSCSI, so no storage integrations. and wondered if the hard Disk read time can be substituted . 8 GB read at 4 MB/s. Slow backup perfomance in HOTADD (virtual appliance) mode This problem is related to the TCP/IP implementation of these arrays and can severely impact the read performance of storage attached to the ESXi/ESX software through the iSCSI 1. In the screenshot we see a very high latency (last column) of seconds (!!!), which is too slow for the mentioned disk type, veeam agent reads the most data of all processes. At some point in the past month just after a client update, the read speed of the C drive has slowed down to ~8MB/s from a previous read speed of 150MB/s+. If the speed between disks is comparable and the overall bottleneck reported by the job is target, then you need to pay attention to your target storage. The 2nd hard disk is 150GB in size its been calculating digests for 7 hours now and is only 3% of the way Veeam B&R 11 Veeam MS Agent 5 vSphere 7. 6GB restored at 349KB/s anbd] 157:xx:xx” Veeam Community discussions and solutions for: 10/26/2021 9:47:09 AM :: Hard disk 2 (800 GB) 630 GB read at 4 MB/s [CBT] 10/28/2021 6:42:38 AM :: Removing VM snapshot that would mean it's the vmkernel port going slow here, so look at which NICs are being used. Asynchronous read operation failed Failed to upload disk. 1. Wondering if my offsite storage wasn't seeded correctly. 0 Update 3 host running on a i7 system with 64GB, SSD and HD storage and a 1G NIC-VBR 12. If I run a full backup the job will show over 1000MB/s read and the repo will show the Nic at 8Gbps coming in. Search. If performance returns to "normal" then at least you know the problem is with the CIFS portions writing to the D2D, if not, then you know it's almost certainly the source data retrieval. viracoribt » Tue Oct 28, 2014 3:09 pm. The backups run in the evening so there won't be user activity on the server. Did a Veeam backup to USB disk, restored at target site. VP, Product Management Posts: 27405 Veeam Community discussions and solutions for: Slow backup (San to San) of Veeam Backup & Replication. m. As noted NBD mode is the primary method used if The backup is on an external hard drive so even if it's using a lot of disk "bandwidth", it's a different disk from where Windows and my apps are, so I don't see how that would slow them down. 9 TB) 1. I believe that he is doing file-based backup (expected to be This is not a support forum! Please do not bring environment-specific issues here but contact our Customer Support for assistance and troubleshooting. 242 for disk Hard disk 2 [san]). Now, being a new veeam user this is what i've collected from reading. Those two VM's always ran Based on this statistics we can say that Source Data Mover spends most time of its activity on reading data because of slow read in NBD mode. 172 of Veeam Backup & Replication. 250. Can you use a Veeam proxy with virtual appliance mode (with NBD disabled) to backup that file server instead? as you are changing the disk configuration for the VM even though you are just adding a new disk. Veeam Community discussions and solutions for: Slow backup performance of Veeam Backup for Microsoft 365. 9 GB) 3. Failed to Open VDDK Disk [is read-only mode] true. I've run sdelete on the disks and performed an active full backup a couple of times, but the Veeam Community discussions and solutions for: Slow replication with 12. I know you mentioned a VeeamZIP backup, but it was slightly less clear what feature you used to restore. 0u1d (soon to be updated to U2)-----ReFS seems to be quite a bit slower than NTFS. Veeam Community discussions and solutions for: Really slow merging to cloud storage of Veeam Backup & Replication. 0 B read at 0 KB/s [CBT]" then after took long time on this step for each disk, the backup finish successfully with processing rate between 50 MB/s and 90 MB/s. [CBT] No errors. 2. So we see that during backup when veeam was reading and writing from and to the same disk, performance was very good. Search Hard disk 1 (40,0 GB) 2,6 GB read at 120 MB/s [CBT] Top. I paused them to set up a few backup jobs. 0 KB read at 0 KB/s [00666177] of VMware vSphere Hard disk 2 (5. Due to this backup window is getting extended. Basically, the bottle next doesn't really lie, imo, and you can just look towards how fast can you do random read on the USB disk. :: Swap file blocks skipped: 44. I have a VM with a 16TB, independent disk. Not a support forum! Skip to content 06/11/2015 18:59:53 :: Hard disk 1 (30. 8GB Summary: Veeam Community discussions and solutions for: (10TB) from the internal hard disk to an SMB share on the NAS. You could try to force network mode for the target proxy and see whether the processing speed becomes better (hotadd can put high I/O load on the target in some cases). This may be suitable for smaller VMs where backup sessions are short due to small vhdx files. Also with changed block tracking entire disk should not be read during incremental job runs. I tested this one(Hot add mode) and dedicate preferred network but it was no different. There is no way the entire contents of the drive changed Now that I look at this it appears for many of my backups Veeam is re-reading the entire contents of the drive as changed. I dont understand why. I do that once every quarter. 5 TB) 1. Here’s a summary: 2 ESX 5. [CBT] DISK2 . Installing a proxy on the VM having access to the datastore with VMs you are backing up, will allow for using hotadd source data retrieval mode, With no throttling, backup read speed fluctuates between around 60Mb/s and 140Mb/s (note - when the read speed was around 77Mb/s, network usage shown in Task Manager on the server computer for Veeam Agent was When I backup, my main (C:) Internal SSD shows: Local SSD (C:) (476. The Last backup took approx 14 hours. Two nights ago, i started noticing that with two VM's (out of 19 being backed-up and 7 replicated), the read-speeds in their jobs, regardless if it's backup or replication, have become horrendously slow. was the bottleneck listed but I'm trying to get the remaining VM's seeded over this long weekend but I'm getting really slow speeds on the current job (8MB/s) and at that rate it will take Which is very very very slow. 4 TB) 54. Since cpu of Veeam server is < 10 % and disk usage is also low we wanted to check Veeam process usage and found out that also Veeam. High read latency on the source disk sub-system 2. It still has to read the original data from the disk, either through the VM snapshot or the storage snapshot. Hard disk 2 (15 TB) 6,2 TB read at 75 MB/s [CBT] 23:59:11 Hard disk 2 (15 TB) 1,5 TB read at 1016 MB/s 27:18 So from slow disks to slow disks with 1016 MB/s that's ten times faster. 5 hours to backup 68GB of data on the file server. Backup file just only 100G, but I often take 4-5 hours to finish restore ( Restore VMs hard disk) This registry entry fix has resolved the speeds from the HP StoreOnce (Disk) to Tape: Key: UseUnbufferedAccess Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Type: REG_DWORD Value: 0 This key must be added on the Veeam Backup & Replication server. There are many reasons why restore can be slower than backup, for example: backup is running in SAN mode and the restore proxy works in Network mode (btw from the log I see it's NBD), slow read from the backup repository (f. . Here the transfer speed does not slow down. 168. Note the copy speed. 0 GB read at 60 MB/s (GBT)" I hadn't yet found an easy way to specify Veeam to make a new full vs an incremental, but deleting all previous full & incremental backups on the external drive in the past had seemed to work? Here is the job summary I've been using. disk c was 52 MB/s, disc d was 13 MB/s and disc e was 9 (!) Hard to tell without actually testing it. Shouldn't occur normally unless you're I have a small virtual setup which I’m backing up using Veeam B&R 7. As I'm type this the job is reading an 800 GB hard drive, 8. We also see that there is a IO of 11 MB/s, which is also very little BUT we see 100 % disk usage (blue rectangle). It just took 5. sysstat -x reports 27% disk utilization at max during VBR backup. Have you tried to reset CBT manually? Do you see the same behavior for all the jobs/VMs? Symptoms: Backup sits at 0KB on the hard disk read step. The target is not there. 0 TB) 806. How can I know where the replica medata are stored ? Issue 2 Source VM disks were not re-sized or re-attached. Disk 0 AVAGO SMC3108 SCSI Disk Device Capacity: 161 TB Formatted: 161 TB System disk: No Page file: No Read speed 82. No writes I am aware of. 0 GB) #### read at 3MB/s [CBT] 18/11/2013 7:23:52 p. Veeam Community discussions and solutions for: SOLVED! Veeam Backup to Synology NAS takes forever of Veeam Agent for Microsoft Windows When I did a full system backup the transfer rate for each disk got "significantly" slower, e. Do you mean the read speed for these disks is slower than for smaller ones? You can see the hard disk read speed in the VM session log. I can't configure that in my opinion. Simply expand ESX and browse to vmfs folder, then copy the file and paste it into "My Computer" local hard drive disk. 9 GB read at 54 MB/s [CBT] One thing to consider is that when you restore from that hybrid Nimble, you're likely doing "cold reads" directly from the SATA disks; that'll slow things down a bit. The read speed will depend on change blocks placement: the more random they are, the slower the changed blocks will be read from source storage. 0 B read at 0 KB/s [CBT]" and hang there for a long time before actually starting. Veeam backups on a very similarly configured R420 and R530 running busy VMs to the same or an identical Synology were fine, about 110-115 MB/s - wire speed, really. The disk is 4Tb Intel P4600 NVMe SSD, pretty fast one, with ~2Tb used in databases and system files. It took over 9 hours! In the quickmigration summary it says the bottleneck is the target. 9 MB/s Write speed 0 KB/s Active time 5% Average response time 0. On disk 1 it reads only the used space, on disk 2 and 3 it READS the entire disks. Data: Processed: 969GB Read: 68. 65TB used in total. 1 GB read at 25 MB/s [CBT] 06/11/2015 19:19:17 :: Using backup proxy VMware Backup Proxy for disk Hard disk 2 [san] 06 According to the bottleneck stats, it is clear that the issue is write speed to the target storage. I am utilizing VMware vSphere VSAN as storage, which is considered as DAS (Direct Attached As to the read rate for each disk in the action column, I would have thought it's just the amount read divided by the duration listed for that disk, but in practice it tends to be a bit higher. The storage can be local disks, directly attached disk based storage (such as USB hard drive), or a iSCSI/FC SAN LUN in case the server is connected into the SAN fabric. Veeam Community discussions and solutions for: Veeam v9 Backup Performance Slow of Veeam Backup & Replication. Hard disk 1 (40. Yesterday we transferred two VMs at a time (srv01 with 1x 40 GB HDD and srv02 with 1x 100 GB HDD + 1x 300 GB HDD). Hard disk 1 (80,0 GB) 9,8 GB read at 30 MB/s [CBT] Top. I could be totally wrong. Only wan acc. And I attach one external HDD to one VM I use Veeam on (1) to backup all VMs and store backup data on external HDD on (2) So my problem: I got terrible speed when I restore VM on (2). In both cases as well the bottle neck is the source. We're evaluating VEEAM Agent for Windows 2. If I recall correctly, I remove the disk from the VM, and then add it back as an existing disk, and once it’s showing correctly, then Veeam can grab the disk for backups. It seems like "Sync time" for each thread has risen significantly, over 30+ minutes, making me wonder if there are some kind If possible I would suggest carving up a local repository of local disk and testing to that. Random jobs, random vm's, same issue. Thanks!. 0 GB) 500. Code: Select all I just can't believe that neither Microsoft, nor Veeam has got no solution for this! We ended up disabling CBT for our high IO VMs, deleting the MRT and RCT files for the virtual hard disks. The images on my first post shows : DISK1 . Veeam Community discussions and solutions for: Hard disk 1 (147,3 GB) 5,7 GB read at 84 MB/s 21. Veeam Community discussions and solutions for: Veeam CBT Processing Full Drive of VMware vSphere. This task reads the entire source VM from disk in order to verify the integrity of the VM data, prior to failing back to it. So here's my question: Copy jobs seem to be very slow, but not as slow as before the optimization we did with Veeam Dev in V11 (case 04822737). R&D Forums. I do have a ticket in with VEEAM support, but they have currently advised me to talk with VMware which I am doing about the issue. Manager. Veeam Community discussions and solutions for: The problem we have however is that a replication done using CBT is slower than doing a full copy, and slower than a replication without CBT enabled. The reason Im doing it is running active fulls is time consuming. On occasion we get disks that just sit at this step for ages (until the job I’m probably going to create a test Veeam job that reads at the file level instead of the volume and see if that makes a difference here in a few minutes. While the proxy is reading the data to back up, the production VM experiences higher I/O latency. -when I do a restore of those VMs, including the Veeam Community discussions and solutions for: Backup copy job is slow of Veeam Backup & Replication. 0. rct files). 0 GB) 26. I've made sure that I have no residuals left behind. iSCSI LUNs are used to This feels like disk IO bottlenecking so @regnor ’s comment around instant recovery makes sense. 9 TB read at 41 MB/s [cbt] - Can anyone tell me how long it should take this disk to complete backups? The backup ran for 13:21 hrs and failed to complete due to backup window setting. Using backup proxy VMware Backup Proxy for disk Hard disk 1 [nbd] Please provide the stats and check if same transport mode is used for those VMs. But if i manually copy the vbk file from the source repository and "paste" it to the usb 3 disk (over the share from the nas) speed is fine. 2 posts • Hi Foggy Bottleneck is Target. 1,3 TB (102,5 GB used) Hard disk 1 (100 GB) 2,2 GB read at 103 MB/s [CBT] Hard disk 2 (300 GB) 0 B read at 0 KB/s [CBT] The corresponding log entries when the entire disks are read Hi All, it’s my first post here. Ranging between 700 and 1000 ms. 0 GB read at 48 MB/s (GBT)" "Hard Disk 2 (1. Our secondary backup proxy takes upwards of 10-20 minutes to start reading data from the hard disks using CBT. Backing up locally to a Windows 2012 Veeam Server with Direct Attached Disks (MD1220) and Direct SAN access is giving me good results BUT when i replicate to our DR site the replication is really slow, even the incrementals is taking hours. I want to replicate VMs to separate dedicated mirrored virtual volumes within the same stretch cluster. Veeam Community discussions and solutions for: Export List of Hard Disk Read Times of PowerShell. SyncDisk}. of Veeam Backup & Replication R&D Forums. 0 GB) 14. Just thought of telling you guys what I'm seeing because, although the issue is exactly the same, we are using another backup solution. Not a support forum! Skip to content I am going to run a backup to a local disk from that server to see what read speeds we get; this will hopefully give better insights on if the read If you edit the VM and look at the properties, does it show the disk properly there? Usually when I see a 0B disk, it’s actually a VM issue and not a Veeam issue. Which is very very very slow. The backups run in the evening Using my SQL server again my last Replica took approx 2-3 minutes for that machine. NBD is what's slowing this down. , Load: Source 63% > Proxy 73% > Network 54% > Target 0%) Also - how loaded is the 10Gb link between sites? Do other systems use it? According to the log you posted, the job is using hot add to backup this particular hard disk. Whatever I do, I cannot reach the FTP-transfer speed of 100MB/s per Harddisk. Every time a synthetic runs im closing that chain of backups, less chance of corruption. 0 GB) 144. Please don't forget to share support case ID. Comprehensive data protection for all workloads It takes 27 minutes to read 7. Vitaliy S. took more than 4 days to complete) I don't know what else to do to get this volume backed up. Dedicated Backup Server (Veeam B and R currently installed on) - HP Proliant D320e Gen 8 - Windows Server 2012 R2 - 16GB RAM - 4 Core - 500GB Hard Disk - Internal Backup Storage - QNAP Turbo Nas TS420U 10TB Offsite External Backup Storage - Currently using Samsung M3 4TB USB 3. Veeam Backup & Replication installs the following components and services: Veeam Installer Service is an auxiliary service that is installed and started on any Windows server once it is added to the list of managed servers in the Veeam Backup & Replication console. Resource Monitor says the disk and network are nowhere near saturated. 0 MB 18/11/2013 7:23:54 p. Our primary Veeam server also has the backup proxy installed and when backing up a virtual machine starts reading the hard disks (vmdk files) within a minute or so of starting the job. 6GB restored at Performance is as good as the weakest link. The type of RAID controller itself can also play a role. 0 GB Veeam Community discussions and solutions for: Job stuck on hard disk read of Veeam Backup & Replication. Even after more than 10 hours, the file is still being written at over 100 MB/s. I’m evaluating Veeam in my lab. 3 GB read at 73 MB/s [CBT] Disk Backup was done from a none deduplicated NetApp volume. 0 TB) 438. Yes, at first, we have RCT enabled in the Veeam backup and found that we would need to live migrate the SQL VM to get the performance back (even after shutting down the server and removing the . What is the expected duration in time per GB for this proces ? Top. Quick links Hard Disk 2 (20. We had an immutable copy at our DR site which has now been copied as the replica for the CDP job however, we are now seeing the VBR server state "Processing disk hard disk x" (0% done Right now that veeam is runnning a full backup and the first 3 disks are backuped, and 2 disk are still being backuped. 2 Disks (100 GB and 4TB), both thin provisioned, 1. 5, vSphere 6. 2020 01:05:24 :: Hard disk 2 (784,2 GB) 14,8 GB read at 3 MB/s Those two lines up there tell me that a the Copy Job processed the second Hard disk 20 times slower than Hard disk 1. No matter what if you have a spinning-disk hard drive you should upgrade to SSD (They start at $15 for SATA SSD). The only latency we see is on the veeam proxy disk statistics. :: Using For some background - The source volume is being read via SAN mode (1/22/2018 6:02:30 PM :: Using source proxy 192. The bottleneck is always displayed as target Datastores ! Deployment is fully automated. VIB or . 0 GB) 45. 0 Disk. Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd] VMware Backup Proxy is "this Server" with Transport mode set to "Automatic" Top. The percent busy number for this component indicates percent of time that the source disk reader spent reading the data from the storage. Since I need to back up the entire VM and run pre/post job scripts, I created a B&R managed agent job for it. Btw, what is the RAID configuration on the target host? Veeam Community discussions and solutions for: Slow Backup Speed (7-30 MB/s) So I suggest you to try hotadd proxy (just install an extra windows VM to both of your hosts). but very high CPU consumption and constant smaller disk read access. ETA: Made a job and told it to only back up OS files and the user folders like Documents and Downloads. Backup. 2. Veeam Community discussions and solutions for: Slow performance using HOTADD mode in VBR 9. Not a support forum! Hard disk 1 (80. Finally got round to upgrading to version 6 of Veeam recently - since then the performance of one of my servers is very slow. our support was able to isolate the issue to VMs with Resilient Changed Tracking (RCT) enabled. foggy Veeam Software Posts: 21140 Using source proxy name1 for disk Hard disk 1 [hotadd] Using target proxy name2 for disk Hard disk 1 [hotadd] A planned failover/back When performing a failback after a "Planned Failover" operation, Veeam requires a task called "Calculating Original Signature Hard Disk" to be performed. So we've opened a case with Microsoft Veeam Community discussions and which are the slow ones. They get read at 1MB/s if i'm lucky. and appears to have re-read the entire hard -NAS is capable of sustained 112MBs read and write speeds. 3 GB read at 70MB/s and it's been working on that for about 1. From previouse restores provided by the windows-built-in backup tool recovery was always very fast and I think this should be the goal too. 0 GB) 99. 8 GB read at 24 MB/s 22:30 Hard disk 2 (1. Not a support forum! Skip to content. Last month I celebrated 15 years of answering "Very slow restore performance from [insert a deduplication appliance name]" type of threads on this forum You asked for a direction so here's my suggestion on what I personally would do. Slow read is SAN mode can have different root causes depending on infrastructure specific and used hardware. 0 KB read at 0 KB/s [00666177] Post by jbarrow. 2 GB read at 13 MB/s 01:14:52 Veeam Community discussions and solutions for: "Read at 1 mb/s" of Servers & Workstations. Hard disk 4 1024 GB | SCSI(0:3) --> raw map device (storage 3par LUN SATA) Veeam Community discussions and solutions for: Very slow backup on simple setup of Servers & Workstations R&D Forums. Veeam Community discussions and solutions for: Direct Storage Access FC 17-03-2016 15:09:13 Restoring Hard disk 1 (95,0 GB): 49,0 GB restored at 132 MB/s That seems slow, but i've read somewhere that theres an issue with that on 3par, and that i need to create a new lun, correct? Veeam Community discussions and solutions for: during the process the action "Creating fingerprints for hard disk (x)" is created. I've contacted support and have spoken to 3 different level 1 techs. Not a support forum! Skip to content Today we fount that one our jobs stuck on hard disk read, duration 438 hours. - Backup storage for the Veeam Backup jobs are external USB hard disks outside the server room, targets are CIFS folders on the external hard disks - Backup storage for the Veeam Replication jobs is another Synology DS1513+, targets are also iSCSI LUNs - Switches are all gigabit per port Veeam Community discussions and solutions for: Backup to Disk - Synthetic Full - very Slow of VMware vSphere. Veeam Community discussions and solutions for: I also noticed on the servers that the DISK for read + write never go over 200MB/s during the replication. The VM has a size of 2TB and as of this writing the “statistics” shows that I have still 900+ GB left with the restoration rate at 2MB/s at 56% in progress state while the “log” shows “Restoring Hard disk 1 (2. Veeam Community discussions and solutions for: Amazingly slow inital replication of VMware vSphere. Quick links. Agent failed to process method {DataTransfer. -when I do a backup of any of the VMs, it runs at about 91MB/s. - The storage copy job I have manually selected the QNAP as the source, and it copying to a local USB hard disk plugged into the backup server for a seed for offsite cloud connect backup. We have similar virtual backup tool where we are utilizing multiple backup thread for single hard disk. Veeam Community discussions and solutions for: Hard disk 2 (5. Exception from server: The device is not ready. After upgrade, such backup repositories will work as backup The difference between the actual disk write rate and the Veeam restore rate (which matched the NIC sent bytes/sec) is, as I said, I believe due to the fact that my backed up data was compressed at 60%+ the network send rate matches perfectly the backup storage disk read rate. Asynchronous Brian, I do not see the [CBT] tag next to the hard disks processing lines, so seems that the entire image is being read, causing the job to run longer. This service analyzes the system, installs and Previous backups sessions are shown under History -> Backup -> Agent tab of Veeam console, however even incremental run should show detected source disk capacity. Preparing backup proxy VMWare backup proxy for hard disk 1 ( nbd) Check the logs here to see what is being reported as well to help narrow down the issue - C:\ProgramData\Veeam\Backup. 1 GB) 262. In our environment we have dual 40 Gbe to the switch and 10Gbe to our repo and multiple sans. I've had some jobs take 15 minutes to start wheras one job has took 40 minutes before starting. When a replication takes place, we see massive write latency on our vm stats for the veeam proxies using hotadd. BUT CBT IS working and Veeam still reads the entire disk. Top (2) I have several ESXI server and install a lot of VMs. it just seems a problem when veeam is copy the files I can only recommend to text every component independently from Veeam 1) test repository write speed 2) test network 3) test source read speed As far as I see, support already did that and figured out that the disk performance is low. The following limitations apply when Veeam Backup & Replication uses Fast Clone for Microsoft Windows or SMB backup repositories: Veeam Backup & Replication does not use Fast Clone for backup repositories configured with Veeam Backup & Replication 9. Normally when doing an incremental backup with change block tracking, processed data is really small (only changed blocks). However, I read that Veeam Replica need at least to keep the two last snapshot for working properly. foggy At random, our backups will become very slow. However, unless I'm mistaken storage integration would only effect the snapshot process. Since 1 Mb/s is quite slow, I'd recommend to contact our support team and to ask our engineers to look for some hints in debug logs. 0 GB) read at 19MB/s (CBT) 25/01/2014 21:37:08 :: Hard Disk 3 (100. It is the VM that contains Veeam which may explain it, but before the upgrade it would run at around 500Mb/s and complete the 150Gb in a matter of minutes - it is now taking nearly 50. Presumably the duration of the "Hard disk" action includes operations that don't count toward the read time, either before or after the actual disk Is this slow? This seems slow to me. It is reading the entire disk data to identify what blocks should be transferred to the target. CID: 00681602 (you should be able to read this next to the disk read speeds) - Bottleneck stats: It will say Busy: Source,Proxy, Network, Target. So Veeam does a full read, and CBT still works but kicks in for the subsequent runs after this one. What could be going on? My jobs ran fine for the last few days. [CBT] DISK3 . Issue ID The bottleneck stats is showing source and yes i do see the CBT tags at the end of the VM Hard Disk "Primary Bottleneck: Source" "Hard Disk 1 (50. 8GB Transfered 46. Veeam Community discussions and solutions for: Fast then slow Backup of Veeam Backup & Replication. Veeam will read backups as fast as your backup storage can serve the requested blocks, simple as that. The main reasons are slow data "Source 99%" means that source storage data retrieval speed is a bottleneck. 5 GB read at 142 MB/s. I benchmarked If you edit the VM and look at the properties, does it show the disk properly there? Usually when I see a 0B disk, it’s actually a VM issue and not a Veeam issue. There are no checkpoints on the host. I find backups are at a fast 93MB/s, but restores are slow at 3MB/s. -ESXi 7. 0 GB) 24. Option 2: Use Network or Virtual Appliance Transport Mode If you'd prefer to restore the disk as Thick (lazy zeroed), performance can be improved by using the "Picky proxy to use" option in the Full VM Restore and Virtual Disk Restore wizards to select either a proxy that does not have Direct SAN capability or a proxy that has been manually configured to use either A transport mode is a method that is used by the Veeam Data Mover to retrieve VM data from the source and write VM data to the target. On the server that backs up ok, the read rate is between 70 - 80MB/s, on the server that is having issues, the read rate is 74KB/s! Read is the content of the VM which the Veeam proxy had to read and analyze. The device is not ready. I cannot cancel the job either. Specifically, it goes extremely slow (slower than restore Backup proxy "vmwareproxy-01" for Hard Disk 1 nbd]. Veeam Server is physical with an iSCSI connection into the iSCSI network and has all mirrored DataCore ESX virtual disks presented to it in Read Only mode. 0 B) 0. We do not receive any notification about problem. You can find more detailed description of bottleneck analysis here, on Veeam Community Forums: READ THIS FIRST : [FAQ] FREQUENTLY ASKED QUESTIONS v6 - R&D Forums. We didn't remove any snapshot on the replica. VBK's from the NAS to Veeambackup server and have acceptable performance. Job efficiency and time required for job completion greatly depend on the transport mode. As you have at least two different tasks running on the disks (replication and copy job), the disks will perform for random access. m1kkel Enthusiast Posts: 47 Liked: 1 time All disks (both on source and target site) in the jobs are shown as HOTADD However its too slow, a 500GB disk took over 10 hours to calculate disk digets. The operation that takes the most time is the Hard Disk backup [CBT], this is the same for the Esxi host which backs up ok, and the one that doesn't. g. One vdisk is 16 drives and runs in raid 6. 5 GB read at 86 MB/s [CBT] Hard disk 2 (50. The disk is at under 50% capacity, and from Task Manager CPU is at ~25%, RAM (32GB) is at 60%, the disk fluctuates between 0% and 20% utilization, with the disk transfer rate at 0, peaking Veeam Community discussions and solutions for: Backup are super slow after upgrading to Version 11. But if we do a FLR or a bare metal recovery, performance will be very bad in the end. The backup window for the job is set for 15 hours 5pm - 8am. If this is the case, try setting the throttling to only allow connection from the Hard disk 20 (3. Yet running backups on larger VMs will take alot more time now. 6/26/2019 2:20:51 AM :: System Reserved (disk Issue1 Yes, we have a Veeam proxy on the target side and it is properly used. storage level deduplication), decompression is slow due to CPU load on the backup repository and many others. mrt and . with 12 disks in a RAID 6 array, presented as a RAW iSCSI target and mapped in VMware as a 25 TB disk attached to the Veeam Backup Proxy server. I wonder why it is doing that as I know for fact that shouldn’t be the In yesterday’s Word from Gostev Forum Digest he stated: “ Important Hyper-V news: for the past few months, we've been working on a strange issue with a few customers: poor I/O performance on VMs protected by Veeam. I had to reboot both the Veeam Backup Server and also the Everything’s functional. The Management Network operates on a 10 gig connection. 0, utilizing both backup jobs and replication jobs. SQL sever involves high iops so the host drives may already Been under load when the vm disks are mounted into the veeam proxy to read. - Using backup proxy **** for disk Hard disk 1 [san/nbd] And also hangs on trying to complete the following subtasks: - Finalizing - Coping data from disks (but showing zeroes on speed and kb/mb read) After some time (usually more than an hour) it continues without errors and eventually the whole task Is complete. We never see any latency on our Dell Compellent SAN, or ESX host/Veeam Proxy CPU and memory. Not a support forum! Hard Disk 1 (100. For example, 99% busy means that All disks are 600gb 10k SAS disks. Bad performance of network channel between ESXi and proxy (likely not the reason in our case) (Hard R&W speed tools) and Hard Write speed on Veeam server is 9Gbps 2. 0, plenty of proxies, and 10GB network. Not a support forum! I can perform Read/Write within Windows with manually copying single or multiple . 1 TB disk, which lasted for more than 8 hours. However, my other Internal SSD ‘terabyte’ (X:) Terabyte (X:) (953. Only if your issue is confirmed to be a bug or a limitation that is worth discussing with the R&D team, you may also post it here according to the following rules: ok. 9 TB read at 63 MB/s (GBT)" "Hard Disk 3 (500. With no throttling, backup read speed fluctuates between around 60Mb/s and 140Mb/s (note - when the read speed was around 77Mb/s, network usage shown in Task Manager on the server The Last backup took approx 14 hours. Plus I don't even see much read/write bandwidth (maybe 15-20 MBps each) in Resource Monitor, just that the "Active time" of the external hard drive is at I'm facing a strange issue with a customer: he has many VM backupped with Veeam, and two of them have big disks (1 TB each) almost empty now. It happened twice in the last 20 days, both times on Monday. I’d also check the hypervisor’s resources, just in case it’s over provisioned, if you restored a new copy of the VM and left the original running without network I'm contacting NAS support and will probably contact Veeam support on this - we are still seeing slow write performance on reverse incremental on the 12 and 16 disk NAS devices in use (right now as low as 4MB/s). It might be slow when hotadding disk when the job start, but disk processing speed will be way faster. 0 GB) 0. Previously when I upgraded from V10 to V11, I had similar issue with the backups from my StoreOnce Repository to Tape where speeds were very slow and the Registry fix I used to resolve the slow backup speeds from the HP StoreOnce (Disk) to Tape: Key: DisableHtAsyncIo Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Type: I'm Currently evaluating Veeam B&R 6. Not a support forum! I seen this code for extracting the Removing Snapshop Statistic. Cause: Issue with the new high-perf backup backup file interaction engine logic that can happen if a backup storage is very slow to respond to a request to open a backup file. That would mean the USB 3. 0 Slimline Portable Hard Drive on a daily rotation. Any advice here is appreciated, because the copy jobs for all my vm's are taking longer than the interval of the normal backup jobs. FAQ; Main 5/23/2016 8:17:49 AM :: Hard disk 2 (1. nettec Novice Posts: 4 You can easily test this with a disk test tool in a VM to see what the read and write maximum speed is. At least when creating the first, full backup and incrementals. exe -SHOWPROXYUSAGES is no longer working Giving what I seeing, I don't think it has to do exclusively with ReFS or Veeam, but it seems to be a bug on how RCT is handling read operations inside the virtual disk. The data files are not there on the target or the 8-12-2013 23:25:29 :: Hard disk 2 (750,0 GB) 8-12-2013 23:25:29 :: Calculating disk 2 digests Is a backup copy job also using the proxy on the other side. X. Running Veeam B&R console on a laptop is ok, but make Sometimes the backup job is very slow because the entire disks are read on one or more vm. From what I have read, Synology products offer the possibility to disable strict allocation via I have setup a new veeam setup mix between physical and virtual servers as follows and max Rate Speed i can see is 27 MB/s with some VMs and some I can see even worse 16 and 18 MB/s One job is running now on VM contains 2 Hard Disks, 1 40 GB Thick and one is 170 GB Thick. 0 GB) read at 23MB/s (CBT) Veeam Community discussions and solutions for: Direct Storage Access FC - slow of VMware vSphere. a file server with 10 separate VMDK files can take hours, at 7MB/s per disk yet a VM with a single disk can transfer at >500MB/s over the same infrastructure - COFC 8GB FC Veeam Community discussions and solutions for: SAN Backup Slow - real slow! of VMware vSphere. Veeam Community discussions and solutions for: >6TB to our DR site however the largest of the drives was taking a significantly long time to read/copy over. davwalbfs Lurker Posts: 1 Liked: never Veeam Support Knowledge Base; Slow backup perfomance in HOTADD (virtual appliance) mode BACK TO KB LIST. Setup replication jobs to replicate the VM's, the jobs run REALLY slowly - eg; hard disk 1 is 15GB, it took 16 hours to calculate digests then 35 minutes to replicate the changes. Logged on to the proxy in the failover DC, to further analyze, I see that the proxy is communicating with the host over the network (I would expected direct disk reads since its HOTADDED??). I'm looking at one right now: Hard Disk 2 (2. In general VDDK libraries which available in Vmware will get merged with Veeam software. The issue has been ongoing, but Early on the disk was running fine for about a month or two, but recently it became slower than dirt. Top. Veeam Community discussions and solutions for: Slow transfer to cloud provider of Veeam Backup & Replication. Veeam take 4 hours to backup each of those two VM, while should be lightning fast because the disk are empty. Hard Disk 1 (150. kbr Enthusiast Especially with hot add, where Veeam is able to read data directly from VMDK by itself, in a fashion that is optimized for It seems those local disks are read by the target proxy via Nbd. 23:39:17 Using backup proxy xxxx for retrieving Hard disk 1 data from storage snapshot on vdesvc01 00:07 23:39:25 Hard disk 1 (50 GB) 338 MB read at 131 MB/s [CBT] 00:08 23:39:42 Using backup proxy xxxx for retrieving Hard disk 2 data from JustinCredible, bottleneck shows the component that most of time was busy versus waiting for other stages of processing. I've noticed the job is creating fingerprints, and then reads the hard disk. Another approach would be to place 10Gb at management interfaces Veeam Community discussions and solutions for: Set up BackupJob and now when first Job is Running Processing rate is extremely slow (11MB/s). We have a proxy on the secondary site, and my guess is, reading backup copies from its locally attached disks and writing them through FC to the storage must be much faster than calculating digests endlessly. I Veeam Community discussions and solutions for: Hard disk fully read or only used quota? of Microsoft Hyper-V R&D Forums. 3GB read at 52MB/s [CBT] A week ago, a similar full backup read (for the same source VM): My replica jobs hang on Hard disk 1 (0 B) 0 B read at 0 KB/s [CBT] They sit there indefinitely. Thanks, Top Veeam started to calculate digests for a 1. Veeam Community discussions and solutions for: "Hard disk 4 (0. 1420 running on an i5 system with 16GB, with two 1G NICs, NIC1 con Hi I've been battling this for around 2 months with HPE and Veeam Tech support One issue was that VM's with many hard disks get really slow e. I make a quick backup and Veeam reads whatever it has to read. I have to reboot the server to get it to clear. I do a second one in succession and 1. X for disk Hard disk 1 [nbd] Hard disk drive 1 (200GB) 3,8GB read at 901KB/s How could I increase this speed? I have 2 sites production and DR PRODUCTION: If you abort the job, the metrics go back to normal. Veeam Community discussions and solutions for: Agent Too Slow of Veeam Agent for Microsoft Windows. Depending on the type of disk (5400/7200 rpm, cache on disk) they may not be a good match for that type of workload. Not a support forum! Hard disk 4 (100,0 GB) 14,7 GB read at 1 MB/s [CBT] This is Veeam B&R 9. cvjuynav vgs wdot clqdk miak thochdh pztcj qthswf aoivbgs vjnac