CBX Shell allows you to see thumbnails inside archives in Windows Explorer
https://github.com/T800G/CBXShell
Or use standalone Peazip software
https://peazip.github.io/
Unfortunately, neither seem to work 100%.
Sources:
http://www.nirmaltv.com/2012/06/04/enable-thumbnail-preview-for-zip-cbz-cbr-and-rar-image-archives/