How did the ancients catalog the scrolls of the library of Alexandria? What was their system?

This is an excellent article on the Library. Scan down to the section on The Collection. I think you will find what you want there.

http://en.wikipedia.org/wiki/Library_of_Alexandria