Waas DRE Cache Full causing file issues....

[bradleyordner]

Hi,

I did some searching on the forums and a few people have run into this issue after upgrading memory. I have not upgraded memory recently and I am getting some users not able to open some file sacross the WAN. I have this error in the Waas Engine -

2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690464: (149272:2) RE File Manager error: filemgr_state::do_io (segments) call pwrite(78, 0x958f1008, 65536, 3897950208): No space left on device
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690432: (149378:2) FileMgr failed in writing data-segment start at id=803859, error=0xfffffc09--Disk is full
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690464: (247774:2) RE File Manager error: filemgr_state::do_io (segments) call pwrite(78, 0x978e1008, 65536, 3934322688): No space left on device
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690432: (247897:2) FileMgr failed in writing data-segment start at id=804414, error=0xfffffc09--Disk is full
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690464: (345843:2) RE File Manager error: filemgr_state::do_io (segments) call pwrite(78, 0x967b1008, 65536, 3951296512): No space left on device
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690432: (345980:2) FileMgr failed in writing data-segment start at id=804673, error=0xfffffc09--Disk is full
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690464: (444583:2) RE File Manager error: filemgr_state::do_io (segments) call pwrite(78, 0x96ce1008, 65536, 3910336512): No space left on device
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690432: (444696:2) FileMgr failed in writing data-segment start at id=804048, error=0xfffffc09--Disk is full
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690464: (542842:2) RE File Manager error: filemgr_state::do_io (segments) call pwrite(78, 0x975e1008, 65536, 3944153088): No space left on device
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690432: (542959:2) FileMgr failed in writing data-segment start at id=804564, error=0xfffffc09--Disk is full
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690464: (640002:2) RE File Manager error: filemgr_state::do_io (segments) call pwrite(78, 0x95db1008, 65536, 3925803008): No space left on device
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690432: (640068:2) FileMgr failed in writing data-segment start at id=804284, error=0xfffffc09--Disk is full
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690464: (738661:2) RE File Manager error: filemgr_state::do_io (segments) call pwrite(78, 0x962c1008, 65536, 3923902464): No space left on device
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690432: (738794:2) FileMgr failed in writing data-segment start at id=804255, error=0xfffffc09--Disk is full
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690464: (838685:2) RE File Manager error: filemgr_state::do_io (segments) call pwrite(78, 0x974b1008, 65536, 3869704192): No space left on device
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690432: (838812:2) FileMgr failed in writing data-segment start at id=803428, error=0xfffffc09--Disk is full
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690464: (939427:2) RE File Manager error: filemgr_state::do_io (segments) call pwrite(78, 0x96631008, 65536, 3927048192): No space left on device
2010 Oct 17 20:09:18 NWASYD01 so_dre: %WAAS-RE-3-690432: (939545:2) FileMgr failed in writing data-segment start at id=804303, error=0xfffffc09--Disk is full
2010 Oct 17 20:09:19 NWASYD01 so_dre: %WAAS-RE-3-690464: (35329:2) RE File Manager error: filemgr_state::do_io (segments) call pwrite(78, 0x96dd1008, 65536, 3904438272): No space left on device
2010 Oct 17 20:09:19 NWASYD01 so_dre: %WAAS-RE-3-690432: (35386:2) FileMgr failed in writing data-segment start at id=803958, error=0xfffffc09--Disk is full

I thought having a almost full Dre cache was a good thing for performance?

[Bhavin Yadav · Accepted solution]

Hi Bradley,

This is definetly the DRE issue due to memory upgrade. Please perform following operation to root out this issue.

wae# disk delete-data-partitions

wae# reload

It may be that the DRE partition size is incorrect for some reason on the WAE. Hence, we need to clean it up and align it properly.

Please note that this will clear out the DRE and CIFS cache on the device.

Defect details: http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?caller=pluginredirector&method=fetchBugDetails&bugId=CSCsy47235

Regards.

PS: Please mark this Answered, if this resolves the issue.

[bradleyordner]

Ok i will try this. Although we have never installed memory into this box, it already came with upgraded memory. The last thing we did was replaced Disk 01 as it was faulty.

Thanks for your quick reply,

Brad

[Bhavin Yadav]

Thanks and keep us posted.

Regards.

[bradleyordner]

Looks good..... Cache is reported as usable -

sh statistics dre

Cache:
    Status: Usable, Oldest Data (age): 21m39s
    Total usable disk size: 116735 MB,  Used: 0.01%
      Hash table RAM  size:    417 MB,  Used: 0.00%

Connections:   Total (cumulative): 533   Active: 12

Encode:
   Overall: msg:       3984, in:   2003 KB, out:   1073 KB, ratio:  46.41%
       DRE: msg:       3644, in:   1942 KB, out:   1741 KB, ratio:  10.32%
DRE Bypass: msg:       1036, in:  62427 B
        LZ: msg:       2955, in:   1293 KB, out:    553 KB, ratio:  57.20%
LZ Bypass: msg:       1029, in:    508 KB
    Avg latency:      0.155 ms     Delayed msg:        599
  Encode th-put:   3244 KB/s
  Message size distribution:
    0-1K=87%  1K-5K=12%  5K-15K=0%  15K-25K=0%  25K-40K=0%  >40K=0%
Decode:
   Overall: msg:       4338, in:   1456 KB, out:   2685 KB, ratio:  45.76%
       DRE: msg:       3873, in:   2249 KB, out:   2547 KB, ratio:  11.70%
DRE Bypass: msg:       1495, in:    137 KB
        LZ: msg:       3326, in:    894 KB, out:   1837 KB, ratio:  51.33%
LZ Bypass: msg:       1012, in:    562 KB
    Avg latency:      0.084 ms
  Decode th-put:   7394 KB/s
  Message size distribution:
    0-1K=84%  1K-5K=13%  5K-15K=1%  15K-25K=0%  25K-40K=0%  >40K=0%

Will monitor for a few days.

Thanks again,

Brad