[Mayan EDMS: 1847] Document Cache

[David Kornahrens]

So, I have moved over my main directory to S3. After confirming it was
working fine, I deleted the "document_storage" contents.

However, I still see 11gb of data in the "./mayan_data/_data/document_cache"
location.

Is this something that can also be moved to S3? My point of the move was
to lower data consumption on my server and move everything to S3. How much
data is stored as page cache in this folder?

Thx,
David

[Roberto Rosario]

Yes, the document cache can also be move to an S3 object storage. But be
aware that moving the document cache to a remote storage can add unforeseen
slowdowns when displaying images.

The document cache stores the image representation of the documents to
speed up display of the thumbnails, previews and OCR. It is keep local by
default to speed up access.

Like the document storage, which is abstracted by
the DOCUMENTS_STORAGE_BACKEND setting, the document cache is abstracted by
the DOCUMENTS_CACHE_STORAGE_BACKEND
setting and move to S3 using the same procedure as moving the document
storage.

Post by David Kornahrens
So, I have moved over my main directory to S3. After confirming it was
working fine, I deleted the "document_storage" contents.
However, I still see 11gb of data in the "./mayan_data/_data/document_cache"
location.
Is this something that can also be moved to S3? My point of the move was
to lower data consumption on my server and move everything to S3. How much
data is stored as page cache in this folder?
Thx,
David

[David Kornahrens]

Roberto,
In what method would I use to move the cache? I think the speed shouldn't
be too much of a problem, being hosted on S3. I'm curious how the cache
works though. Does it only store "X" amount of data in the cache? Or does
it basically keep everything in there forever? It's about the same size as
my object storage folder, so I'm not seeing what the cap would be for the
cache folder.

Digital Ocean droplets aren't that large and until they roll out their
object storage, it's really not going to work, so I'll need to put
everything on S3 using the DJANGO-STORAGES option to keep our Mayan up and
running and keep adding files.

Thanks,
David

[Roberto Rosario]

The cache can be moved to S3 with the same process used to move the
document storage to S3.
In version 2.5.2 support was added to disable the cache, partially or
completely.
Limiting the cache to a certain amount of space is a good idea, and a good
solution without having to migrate the cache to S3. I'll add an issue to
track this.

Post by David Kornahrens
Roberto,
In what method would I use to move the cache? I think the speed shouldn't
be too much of a problem, being hosted on S3. I'm curious how the cache
works though. Does it only store "X" amount of data in the cache? Or does
it basically keep everything in there forever? It's about the same size as
my object storage folder, so I'm not seeing what the cap would be for the
cache folder.
Digital Ocean droplets aren't that large and until they roll out their
object storage, it's really not going to work, so I'll need to put
everything on S3 using the DJANGO-STORAGES option to keep our Mayan up and
running and keep adding files.
Thanks,
David

[David Kornahrens]

Robert,
What method would I need to use to limit the cache? Also, what can I do to
flush the cache that is currently built up? We store all files on S3, but
our disk is still growing.

THanks,
David

Post by Roberto Rosario
The cache can be moved to S3 with the same process used to move the
document storage to S3.
In version 2.5.2 support was added to disable the cache, partially or
completely.
Limiting the cache to a certain amount of space is a good idea, and a good
solution without having to migrate the cache to S3. I'll add an issue to
track this.

Post by David Kornahrens
Roberto,
In what method would I use to move the cache? I think the speed
shouldn't be too much of a problem, being hosted on S3. I'm curious how
the cache works though. Does it only store "X" amount of data in the
cache? Or does it basically keep everything in there forever? It's about
the same size as my object storage folder, so I'm not seeing what the cap
would be for the cache folder.
Digital Ocean droplets aren't that large and until they roll out their
object storage, it's really not going to work, so I'll need to put
everything on S3 using the DJANGO-STORAGES option to keep our Mayan up and
running and keep adding files.
Thanks,
David