YACReader Forum: LibraryServer freezes with 25600 byte library.ydb (1/2)

TOPIC: LibraryServer freezes with 25600 byte library.ydb

LibraryServer freezes with 25600 byte library.ydb 6 years 1 month ago #3701

nstephenh

Offline

Fresh Boarder

Posts: 7

Karma: 0

I'm attempting to use the docker image of YACReaderLibraryServer with my library of ~9 tb of comics, but whenever I try to create a library, it freezes. The issue could potentially be related to having the comics directory mounted over NFS, or just the large size of the library. Either way, I had the command running overnight with no change to the size of the ydb file and nothing in the covers directory.

Is there by chance an undocumented verbose option on YACReaderLibraryServer that I could use to see if the program is frozen or just taking a really long time?

Thanks

The administrator has disabled public write access.

LibraryServer freezes with 25600 byte library.ydb 6 years 1 month ago #3702

Luis Ángel

Offline

Administrator

Posts: 2616

Thank you received: 543

Karma: 22

I always recomend to create the initial library using a pc/mac because the CPU makes a huge difference, specially for a library so huge.

YACReaderLibrary will let you stop the process at any moment and resume later. If you have a chance to connect your storage device directly to your computer please do it and then create the library, after that you can just add the library using YACReaderLibraryServer.

Anyway, no covers in the covers directory sounds bad...Could you try to create a library in a small folder? Create a folder in the same path you are tryting to create your library, something like MyComics/test, and add a couple of subfolders there and throw a couple of comics in each folder, lets see if your setup works that way and we'll look into the huge library handling later.

Contribute to the project becoming a patron: www.patreon.com/yacreader
You can also donate via Pay-Pal: www.paypal.com/donate?business=5TAMNQCDD...e=Support+YACReader\

The administrator has disabled public write access.

LibraryServer freezes with 25600 byte library.ydb 6 years 1 month ago #3703

selmf

Offline

Developer

Posts: 631

Thank you received: 135

Karma: 18

Hi nstephenh,

the Docker image is supplied by the community, so I don't know which version of the headless library server it is using. Older versions of the headless server suffer from the absence of even basic path checking, which can lead to empty/broken libraries.

Also, you write that you have 9 (!) Terabytes (!!) of comics mounted via NFS(!!!). That probably is the largest collection I have heard of until now and it's way beyond the largest size we ever tested. I'm not sure if leaving the server running for one night will be enough in this case as the data has to be transferred back and forth for processing.

There is also one known bug related to network file systems and YACReader Library or rather it's database backend, sqlite. Sqlite requires file locking to work and this is poorly supported by some NFS implementations. A solution to this problem can be found somewhere here in the forum, use the search function

Best regards,

selmf

My answers are not necessarily official YACReader statements but mostly represent my own opinion in technical matters.

The administrator has disabled public write access.

LibraryServer freezes with 25600 byte library.ydb 6 years 1 month ago #3745

nstephenh

Offline

Fresh Boarder

Posts: 7

Karma: 0

Sorry its taken me so long to reply. Let me preface by saying I appear to have gotten everything working by

Compiling everything myself (using the latest version as of March 10
Not mounting over NFS
Not trying to figure out docker at the same time

The library is now being created (~10k covers in 12 hours, with other things hitting the disk), and I will test to see if the webui works after I finish.
Luis Ángel wrote:

I always recomend to create the initial library using a pc/mac because the CPU makes a huge difference, specially for a library so huge.

Perhaps if I was using a raspberry pi, but my server is somewhat comparable to my laptop and I can't move the storage so this is the most efficient.

selmf wrote:

Also, you write that you have 9 (!) Terabytes (!!) of comics mounted via NFS(!!!). That probably is the largest collection I have heard of until now and it's way beyond the largest size we ever tested. I'm not sure if leaving the server running for one night will be enough in this case as the data has to be transferred back and forth for processing.

I agree that its quite large. I'm not sure what factor was the one preventing it from loading, probably some combination of NFS, and an older docker image. I imagine that scanning it all via NFS would take forever given that in 12 hours I've scanned 10k out of 230k. I estimate it will take ~12 days at this speed.

Now my only question is can I run library-add and start while library-create is underway? I'd like to test if I can with what I have now, but I don't want to risk breaking everything.

The administrator has disabled public write access.

LibraryServer freezes with 25600 byte library.ydb 6 years 1 month ago #3746

nstephenh

Offline

Fresh Boarder

Posts: 7

Karma: 0

Sorry its taken me so long to reply. Let me preface by saying I appear to have gotten everything working by

Compiling everything myself (using the latest version as of March 10
Not mounting over NFS
Not trying to figure out docker at the same time

The library is now being created (~10k covers in 12 hours, with other things hitting the disk), and I will test to see if the webui works after I finish.
Luis Ángel wrote:

I always recomend to create the initial library using a pc/mac because the CPU makes a huge difference, specially for a library so huge.

Perhaps if I was using a raspberry pi, but my server is somewhat comparable to my laptop and I can't move the storage so this is the most efficient.

selmf wrote:

Also, you write that you have 9 (!) Terabytes (!!) of comics mounted via NFS(!!!). That probably is the largest collection I have heard of until now and it's way beyond the largest size we ever tested. I'm not sure if leaving the server running for one night will be enough in this case as the data has to be transferred back and forth for processing.

I agree that its quite large. I'm not sure what factor was the one preventing it from loading, probably some combination of NFS, and an older docker image. I imagine that scanning it all via NFS would take forever given that in 12 hours I've scanned 10k out of 230k. I estimate it will take ~12 days at the current speed.

I will report back in 12 days and let you know how YAC handles the library. I tried adding the library and starting the server while the scan was taking place but clicking the library at the webui would just take me back to the library selection page.

I also have a question: What do the dots printing to the console mean? Do they signify anything or are they just printed every number of books or time period?

Thanks

The administrator has disabled public write access.

LibraryServer freezes with 25600 byte library.ydb 6 years 1 month ago #3747

selmf

Offline

Developer

Posts: 631

Thank you received: 135

Karma: 18

If I remember correctly the dots on the console are indicators which tell you that the library is being processed.

With such huge datasets you usually have two factors that limit your speed - one is filesystem input/output, the other is CPU power. In your case the limiting factor is probably the filesystem, but, to be fair, it also has to be noted that YACReader is not properly taking advantage of modern multicore CPUs when processing libraries. The whole process is essentially single core single thread, so we're wasting ressources. Changing that is on our TODO list and it will probably happen sometime in the 9.x series, but getting the implementation "just right" is a delicate issue. Your huge collection probably would be a perfect testcase once we've reached a point where a multithreaded library processor is usable …

My answers are not necessarily official YACReader statements but mostly represent my own opinion in technical matters.

The administrator has disabled public write access.

LibraryServer freezes with 25600 byte library.ydb 6 years 1 month ago #3748

nstephenh Offline Fresh Boarder Posts: 7 Karma: 0	I'm 100% certain the bottleneck is IO. My setup is a JBOD array of desktop hard disks and I have other services hitting the disks as well, so performance is limited. Perhaps once some of the other services finish speeds will pick up for me. Thanks for being so responsive. Sorry about the double-posting, I'm on a bad connection (traveling) so I assumed the first one didn't submit properly.
	The administrator has disabled public write access.

LibraryServer freezes with 25600 byte library.ydb 6 years 1 month ago #3749

Luis Ángel

Offline

Administrator

Posts: 2616

Thank you received: 543

Karma: 22

I think that doing some commits during the library creation is needed, otherwise any problem with the library creation will cause a rollback in the DB and all the progress will be lost. I will try to do this for the next release.

There is a new setting in 9.0 for improving performace while browsing remotelly, you'll probably have to enable it until The new server 2.0 will be released. There are some big performance improvements coming.

Thanks for the feedback!

Contribute to the project becoming a patron: www.patreon.com/yacreader
You can also donate via Pay-Pal: www.paypal.com/donate?business=5TAMNQCDD...e=Support+YACReader\

The administrator has disabled public write access.

LibraryServer freezes with 25600 byte library.ydb 6 years 1 month ago #3764

nstephenh

Offline

Fresh Boarder

Posts: 7

Karma: 0

Library finished scanning much faster than projected (probably because other applications stopped hitting the IO.) How do I enable the performance setting? The webui loads and on a cursory glance everything seems fine, but pages take a while to load (Fire tells me that page 3 took 2.8 minutes to load). Also, the page shortcuts at the bottom of the page don't look great beyond 2 digit numbers (and 4 digit numbers are particularly bad). Looking at what was spit out on the console, quite a few files threw parse-rar errors. In total 197137 files were found, and I feel this is short based on other scripts i've run but bash tells me I have 404,814 files (which seems high to me).

Biggest feature request for me would be a search function. With a collection this large, if I can't search then I won't be able to find anything, particularly given the long page loading time.

Also, is there any way to use the YACReader Library on my desktop as a frontend? I don't see one jumping out at me but I could have overlooked it.

The administrator has disabled public write access.

LibraryServer freezes with 25600 byte library.ydb 6 years 1 month ago #3765

Luis Ángel Offline Administrator Posts: 2616 Thank you received: 543 Karma: 22	You have to write the following config in the server config file: YACReaderLibrary.ini which I always forget where is in Linux, it is in this path I think: ~/.local/share/YACReader/YACReaderLibrary/YACReaderLibrary.ini config: [libraryConfig] REMOTE_BROWSE_PERFORMANCE_WORKAROUND=true
	Contribute to the project becoming a patron: www.patreon.com/yacreader You can also donate via Pay-Pal: www.paypal.com/donate?business=5TAMNQCDD...e=Support+YACReader\ The administrator has disabled public write access.