Task #11949 (closed)
Reduce memory pressure in full text FileParser
| Reported by: | jballanco-x | Owned by: | jballanco-x |
|---|---|---|---|
| Priority: | major | Milestone: | 5.0.0 |
| Component: | General | Version: | 4.4.10 |
| Keywords: | full-text indexing, search | Cc: | |
| Resources: | n.a. | Referenced By: | n.a. |
| References: | n.a. | Remaining Time: | 0.0d |
| Sprint: | n.a. |
Description
While attempting to resolve issues with heap OOM errors in the search indexer, multiple stack traces point towards the root cause being somewhere near FileParser?.doParse (see attachment). Additionally, with a large number of text file attachments on a single image (> 1000), if heap size is increased to avoid OOM issues, a new error occurs caused by too many files being open (see second attachment).
Attachments (2)
Change History (6)
Changed 10 years ago by jballanco-x
comment:1 Changed 10 years ago by jballanco-x
Pull request submitted: https://github.com/openmicroscopy/openmicroscopy/pull/2059
comment:2 Changed 10 years ago by jballanco-x
- Milestone changed from 5.0.1 to 5.0.0
This issue is being caused by the current strategy of not closing indexed files until a full batch completes. This is causing not only heap OOM, but under certain circumstances can also lead to an exhaustion of file descriptors. As such, it would be preferable to get this fix in to 5.0.0.
comment:3 Changed 10 years ago by jamoore
- Resolution set to fixed
- Status changed from new to closed
Pr merged.
comment:4 Changed 10 years ago by Josh Moore <josh@…>
- Remaining Time set to 0
(In [c930f183e2c8305bdd64d6c0d23573ce9c017e13/ome.git] on branch develop) Merge pull request #2141 from jballanc/rebased/develop/close-parsed-files
Close parsed files after each indexing iteration (fixes #11949) (rebased onto develop)
Stack trace from an Out Of Memory error in omero.indexer